2023-04-23 05:53:37,017 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4 2023-04-23 05:53:37,030 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-23 05:53:37,048 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-23 05:53:37,048 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/cluster_e3def1fe-aef0-d9be-2e96-db5ec2594b54, deleteOnExit=true 2023-04-23 05:53:37,049 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-23 05:53:37,049 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/test.cache.data in system properties and HBase conf 2023-04-23 05:53:37,050 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/hadoop.tmp.dir in system properties and HBase conf 2023-04-23 05:53:37,051 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/hadoop.log.dir in system properties and HBase conf 2023-04-23 05:53:37,051 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-23 05:53:37,052 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-23 05:53:37,052 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-23 05:53:37,173 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-23 05:53:37,592 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-23 05:53:37,596 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-23 05:53:37,596 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-23 05:53:37,596 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-23 05:53:37,597 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-23 05:53:37,597 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-23 05:53:37,597 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-23 05:53:37,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-23 05:53:37,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-23 05:53:37,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-23 05:53:37,598 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/nfs.dump.dir in system properties and HBase conf 2023-04-23 05:53:37,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/java.io.tmpdir in system properties and HBase conf 2023-04-23 05:53:37,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-23 05:53:37,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-23 05:53:37,599 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-23 05:53:38,248 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-23 05:53:38,253 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-23 05:53:38,538 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-23 05:53:38,729 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-23 05:53:38,753 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-23 05:53:38,806 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-23 05:53:38,842 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/java.io.tmpdir/Jetty_localhost_34111_hdfs____.4wx1l2/webapp 2023-04-23 05:53:38,984 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34111 2023-04-23 05:53:39,002 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-23 05:53:39,002 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-23 05:53:39,591 WARN [Listener at localhost/44199] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-23 05:53:39,678 WARN [Listener at localhost/44199] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-23 05:53:39,704 WARN [Listener at localhost/44199] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-23 05:53:39,712 INFO [Listener at localhost/44199] log.Slf4jLog(67): jetty-6.1.26 2023-04-23 05:53:39,722 INFO [Listener at localhost/44199] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/java.io.tmpdir/Jetty_localhost_37667_datanode____.2vu612/webapp 2023-04-23 05:53:39,832 INFO [Listener at localhost/44199] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37667 2023-04-23 05:53:40,237 WARN [Listener at localhost/41845] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-23 05:53:40,851 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2eeec57cfad746ab: Processing first storage report for DS-75cf324d-79ef-45be-8e7d-e86b031df2fc from datanode 4e0a1e55-43d6-49e6-9e0b-0c82420ce4fb 2023-04-23 05:53:40,852 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2eeec57cfad746ab: from storage DS-75cf324d-79ef-45be-8e7d-e86b031df2fc node DatanodeRegistration(127.0.0.1:39671, datanodeUuid=4e0a1e55-43d6-49e6-9e0b-0c82420ce4fb, infoPort=36309, infoSecurePort=0, ipcPort=41845, storageInfo=lv=-57;cid=testClusterID;nsid=253865043;c=1682229218324), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-23 05:53:40,853 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2eeec57cfad746ab: Processing first storage report for DS-56cd49b1-dd6f-4a9a-b992-8547ffb41e58 from datanode 4e0a1e55-43d6-49e6-9e0b-0c82420ce4fb 2023-04-23 05:53:40,853 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2eeec57cfad746ab: from storage DS-56cd49b1-dd6f-4a9a-b992-8547ffb41e58 node DatanodeRegistration(127.0.0.1:39671, datanodeUuid=4e0a1e55-43d6-49e6-9e0b-0c82420ce4fb, infoPort=36309, infoSecurePort=0, ipcPort=41845, storageInfo=lv=-57;cid=testClusterID;nsid=253865043;c=1682229218324), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-23 05:53:40,880 DEBUG [Listener at localhost/41845] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4 2023-04-23 05:53:40,943 INFO [Listener at localhost/41845] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/cluster_e3def1fe-aef0-d9be-2e96-db5ec2594b54/zookeeper_0, clientPort=63802, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/cluster_e3def1fe-aef0-d9be-2e96-db5ec2594b54/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/cluster_e3def1fe-aef0-d9be-2e96-db5ec2594b54/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-23 05:53:40,960 INFO [Listener at localhost/41845] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63802 2023-04-23 05:53:40,967 INFO [Listener at localhost/41845] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-23 05:53:40,969 INFO [Listener at localhost/41845] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-23 05:53:41,647 INFO [Listener at localhost/41845] util.FSUtils(471): Created version file at hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965 with version=8 2023-04-23 05:53:41,647 INFO [Listener at localhost/41845] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/hbase-staging 2023-04-23 05:53:41,979 INFO [Listener at localhost/41845] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-23 05:53:42,455 INFO [Listener at localhost/41845] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-23 05:53:42,488 INFO [Listener at localhost/41845] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-23 05:53:42,489 INFO [Listener at localhost/41845] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-23 05:53:42,489 INFO [Listener at localhost/41845] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-23 05:53:42,489 INFO [Listener at localhost/41845] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-23 05:53:42,489 INFO [Listener at localhost/41845] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-23 05:53:42,647 INFO [Listener at localhost/41845] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-23 05:53:42,720 DEBUG [Listener at localhost/41845] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-23 05:53:42,815 INFO [Listener at localhost/41845] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44901 2023-04-23 05:53:42,826 INFO [Listener at localhost/41845] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-23 05:53:42,828 INFO [Listener at localhost/41845] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-23 05:53:42,851 INFO [Listener at localhost/41845] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44901 connecting to ZooKeeper ensemble=127.0.0.1:63802 2023-04-23 05:53:42,898 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:449010x0, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-23 05:53:42,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:44901-0x101af94c0060000 connected 2023-04-23 05:53:42,930 DEBUG [Listener at localhost/41845] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-23 05:53:42,931 DEBUG [Listener at localhost/41845] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-23 05:53:42,935 DEBUG [Listener at localhost/41845] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-23 05:53:42,944 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44901 2023-04-23 05:53:42,945 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44901 2023-04-23 05:53:42,945 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44901 2023-04-23 05:53:42,948 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44901 2023-04-23 05:53:42,951 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44901 2023-04-23 05:53:42,957 INFO [Listener at localhost/41845] master.HMaster(444): hbase.rootdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965, hbase.cluster.distributed=false 2023-04-23 05:53:43,026 INFO [Listener at localhost/41845] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-23 05:53:43,027 INFO [Listener at localhost/41845] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-23 05:53:43,027 INFO [Listener at localhost/41845] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-23 05:53:43,027 INFO [Listener at localhost/41845] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-23 05:53:43,027 INFO [Listener at localhost/41845] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-23 05:53:43,027 INFO [Listener at localhost/41845] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-23 05:53:43,032 INFO [Listener at localhost/41845] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-23 05:53:43,035 INFO [Listener at localhost/41845] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40995 2023-04-23 05:53:43,037 INFO [Listener at localhost/41845] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-23 05:53:43,044 DEBUG [Listener at localhost/41845] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-23 05:53:43,045 INFO [Listener at localhost/41845] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-23 05:53:43,048 INFO [Listener at localhost/41845] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-23 05:53:43,049 INFO [Listener at localhost/41845] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40995 connecting to ZooKeeper ensemble=127.0.0.1:63802 2023-04-23 05:53:43,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:409950x0, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-23 05:53:43,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40995-0x101af94c0060001 connected 2023-04-23 05:53:43,054 DEBUG [Listener at localhost/41845] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-23 05:53:43,056 DEBUG [Listener at localhost/41845] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-23 05:53:43,057 DEBUG [Listener at localhost/41845] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-23 05:53:43,057 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40995 2023-04-23 05:53:43,057 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40995 2023-04-23 05:53:43,060 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40995 2023-04-23 05:53:43,064 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40995 2023-04-23 05:53:43,064 DEBUG [Listener at localhost/41845] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40995 2023-04-23 05:53:43,067 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:53:43,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-23 05:53:43,077 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:53:43,097 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-23 05:53:43,097 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-23 05:53:43,097 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:53:43,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-23 05:53:43,099 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,44901,1682229221800 from backup master directory 2023-04-23 05:53:43,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-23 05:53:43,103 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:53:43,103 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-23 05:53:43,103 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-23 05:53:43,104 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:53:43,107 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-23 05:53:43,108 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-23 05:53:43,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/hbase.id with ID: 87f17674-3dd9-4602-9d96-9c120820c8e6 2023-04-23 05:53:43,254 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-23 05:53:43,270 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:53:43,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x68e6aadc to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:43,352 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@251497e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:43,378 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:53:43,380 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-23 05:53:43,401 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-23 05:53:43,402 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-23 05:53:43,404 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:53:43,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:53:43,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-23 05:53:43,449 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/data/master/store-tmp 2023-04-23 05:53:43,884 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:53:43,884 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-23 05:53:43,884 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-23 05:53:43,884 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-23 05:53:43,884 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-23 05:53:43,885 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-23 05:53:43,885 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-23 05:53:43,885 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-23 05:53:43,887 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/WALs/jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:53:43,910 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44901%2C1682229221800, suffix=, logDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/WALs/jenkins-hbase4.apache.org,44901,1682229221800, archiveDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/oldWALs, maxLogs=10 2023-04-23 05:53:43,972 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-75cf324d-79ef-45be-8e7d-e86b031df2fc,DISK] 2023-04-23 05:53:43,981 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:53:44,046 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/WALs/jenkins-hbase4.apache.org,44901,1682229221800/jenkins-hbase4.apache.org%2C44901%2C1682229221800.1682229223922 2023-04-23 05:53:44,047 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39671,DS-75cf324d-79ef-45be-8e7d-e86b031df2fc,DISK]] 2023-04-23 05:53:44,048 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:53:44,048 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:53:44,052 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-23 05:53:44,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-23 05:53:44,112 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-23 05:53:44,119 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-23 05:53:44,142 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-23 05:53:44,158 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:44,164 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-23 05:53:44,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-23 05:53:44,185 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-23 05:53:44,190 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-23 05:53:44,191 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63079993, jitterRate=-0.0600348562002182}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-23 05:53:44,191 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-23 05:53:44,193 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-23 05:53:44,213 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-23 05:53:44,213 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-23 05:53:44,215 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-23 05:53:44,217 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-23 05:53:44,250 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-04-23 05:53:44,250 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-23 05:53:44,277 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-23 05:53:44,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-23 05:53:44,309 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-23 05:53:44,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-23 05:53:44,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-23 05:53:44,321 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-23 05:53:44,327 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-23 05:53:44,330 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:53:44,332 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-23 05:53:44,333 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-23 05:53:44,349 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-23 05:53:44,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-23 05:53:44,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:53:44,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-23 05:53:44,356 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,44901,1682229221800, sessionid=0x101af94c0060000, setting cluster-up flag (Was=false) 2023-04-23 05:53:44,374 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:53:44,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-23 05:53:44,382 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:53:44,388 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:53:44,394 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-23 05:53:44,395 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:53:44,398 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.hbase-snapshot/.tmp 2023-04-23 05:53:44,477 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(952): ClusterId : 87f17674-3dd9-4602-9d96-9c120820c8e6 2023-04-23 05:53:44,488 DEBUG [RS:0;jenkins-hbase4:40995] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-23 05:53:44,500 DEBUG [RS:0;jenkins-hbase4:40995] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-23 05:53:44,500 DEBUG [RS:0;jenkins-hbase4:40995] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-23 05:53:44,504 DEBUG [RS:0;jenkins-hbase4:40995] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-23 05:53:44,505 DEBUG [RS:0;jenkins-hbase4:40995] zookeeper.ReadOnlyZKClient(139): Connect 0x355c4049 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:44,514 DEBUG [RS:0;jenkins-hbase4:40995] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b9ba892, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:44,515 DEBUG [RS:0;jenkins-hbase4:40995] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3fa42b39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-23 05:53:44,541 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-23 05:53:44,547 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:40995 2023-04-23 05:53:44,551 INFO [RS:0;jenkins-hbase4:40995] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-23 05:53:44,551 INFO [RS:0;jenkins-hbase4:40995] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-23 05:53:44,551 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1023): About to register with Master. 2023-04-23 05:53:44,554 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,44901,1682229221800 with isa=jenkins-hbase4.apache.org/172.31.14.131:40995, startcode=1682229223026 2023-04-23 05:53:44,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-23 05:53:44,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-23 05:53:44,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-23 05:53:44,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-23 05:53:44,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-23 05:53:44,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-23 05:53:44,556 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,561 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682229254561 2023-04-23 05:53:44,564 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-23 05:53:44,567 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-23 05:53:44,568 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-23 05:53:44,574 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-23 05:53:44,578 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-23 05:53:44,579 DEBUG [RS:0;jenkins-hbase4:40995] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-23 05:53:44,586 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-23 05:53:44,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-23 05:53:44,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-23 05:53:44,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-23 05:53:44,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,590 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-23 05:53:44,592 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-23 05:53:44,592 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-23 05:53:44,598 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-23 05:53:44,599 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-23 05:53:44,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682229224601,5,FailOnTimeoutGroup] 2023-04-23 05:53:44,610 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682229224603,5,FailOnTimeoutGroup] 2023-04-23 05:53:44,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-23 05:53:44,613 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,613 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,659 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55095, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-23 05:53:44,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:44,689 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965 2023-04-23 05:53:44,689 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44199 2023-04-23 05:53:44,689 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-23 05:53:44,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-23 05:53:44,696 DEBUG [RS:0;jenkins-hbase4:40995] zookeeper.ZKUtil(162): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:44,696 WARN [RS:0;jenkins-hbase4:40995] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-23 05:53:44,697 INFO [RS:0;jenkins-hbase4:40995] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-23 05:53:44,697 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/WALs/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:44,699 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,40995,1682229223026] 2023-04-23 05:53:44,710 DEBUG [RS:0;jenkins-hbase4:40995] zookeeper.ZKUtil(162): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:44,724 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-23 05:53:44,734 INFO [RS:0;jenkins-hbase4:40995] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-23 05:53:44,760 INFO [RS:0;jenkins-hbase4:40995] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-23 05:53:44,764 INFO [RS:0;jenkins-hbase4:40995] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-23 05:53:44,765 INFO [RS:0;jenkins-hbase4:40995] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,766 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-23 05:53:44,779 INFO [RS:0;jenkins-hbase4:40995] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,779 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,780 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,780 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,780 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,780 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,781 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-23 05:53:44,781 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,781 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,781 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,781 DEBUG [RS:0;jenkins-hbase4:40995] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-23 05:53:44,782 INFO [RS:0;jenkins-hbase4:40995] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,782 INFO [RS:0;jenkins-hbase4:40995] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,782 INFO [RS:0;jenkins-hbase4:40995] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,799 INFO [RS:0;jenkins-hbase4:40995] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-23 05:53:44,802 INFO [RS:0;jenkins-hbase4:40995] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40995,1682229223026-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:44,820 INFO [RS:0;jenkins-hbase4:40995] regionserver.Replication(203): jenkins-hbase4.apache.org,40995,1682229223026 started 2023-04-23 05:53:44,820 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,40995,1682229223026, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:40995, sessionid=0x101af94c0060001 2023-04-23 05:53:44,820 DEBUG [RS:0;jenkins-hbase4:40995] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-23 05:53:44,820 DEBUG [RS:0;jenkins-hbase4:40995] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:44,821 DEBUG [RS:0;jenkins-hbase4:40995] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40995,1682229223026' 2023-04-23 05:53:44,821 DEBUG [RS:0;jenkins-hbase4:40995] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:44,822 DEBUG [RS:0;jenkins-hbase4:40995] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:44,822 DEBUG [RS:0;jenkins-hbase4:40995] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-23 05:53:44,822 DEBUG [RS:0;jenkins-hbase4:40995] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-23 05:53:44,822 DEBUG [RS:0;jenkins-hbase4:40995] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:44,823 DEBUG [RS:0;jenkins-hbase4:40995] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,40995,1682229223026' 2023-04-23 05:53:44,823 DEBUG [RS:0;jenkins-hbase4:40995] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-23 05:53:44,823 DEBUG [RS:0;jenkins-hbase4:40995] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-23 05:53:44,824 DEBUG [RS:0;jenkins-hbase4:40995] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-23 05:53:44,824 INFO [RS:0;jenkins-hbase4:40995] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-23 05:53:44,824 INFO [RS:0;jenkins-hbase4:40995] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-23 05:53:44,935 INFO [RS:0;jenkins-hbase4:40995] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40995%2C1682229223026, suffix=, logDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/WALs/jenkins-hbase4.apache.org,40995,1682229223026, archiveDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/oldWALs, maxLogs=32 2023-04-23 05:53:44,955 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-75cf324d-79ef-45be-8e7d-e86b031df2fc,DISK] 2023-04-23 05:53:44,960 INFO [RS:0;jenkins-hbase4:40995] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/WALs/jenkins-hbase4.apache.org,40995,1682229223026/jenkins-hbase4.apache.org%2C40995%2C1682229223026.1682229224938 2023-04-23 05:53:44,961 DEBUG [RS:0;jenkins-hbase4:40995] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39671,DS-75cf324d-79ef-45be-8e7d-e86b031df2fc,DISK]] 2023-04-23 05:53:45,028 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-23 05:53:45,029 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-23 05:53:45,029 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965 2023-04-23 05:53:45,448 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:53:45,451 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-23 05:53:45,454 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/info 2023-04-23 05:53:45,456 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-23 05:53:45,457 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:45,457 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-23 05:53:45,460 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/rep_barrier 2023-04-23 05:53:45,461 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-23 05:53:45,462 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:45,463 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-23 05:53:45,465 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/table 2023-04-23 05:53:45,466 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-23 05:53:45,467 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:45,468 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740 2023-04-23 05:53:45,470 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740 2023-04-23 05:53:45,474 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:53:45,476 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-23 05:53:45,480 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-23 05:53:45,481 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62670993, jitterRate=-0.06612943112850189}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:53:45,481 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-23 05:53:45,481 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-23 05:53:45,481 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-23 05:53:45,481 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-23 05:53:45,481 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-23 05:53:45,482 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-23 05:53:45,482 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-23 05:53:45,483 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-23 05:53:45,488 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-23 05:53:45,488 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-23 05:53:45,497 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-23 05:53:45,511 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-23 05:53:45,513 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-23 05:53:45,665 DEBUG [jenkins-hbase4:44901] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-23 05:53:45,669 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40995,1682229223026, state=OPENING 2023-04-23 05:53:45,676 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-23 05:53:45,677 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:53:45,678 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-23 05:53:45,680 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:53:45,865 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:45,867 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-23 05:53:45,871 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-23 05:53:45,881 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-23 05:53:45,882 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-23 05:53:45,886 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40995%2C1682229223026.meta, suffix=.meta, logDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/WALs/jenkins-hbase4.apache.org,40995,1682229223026, archiveDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/oldWALs, maxLogs=32 2023-04-23 05:53:45,904 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39671,DS-75cf324d-79ef-45be-8e7d-e86b031df2fc,DISK] 2023-04-23 05:53:45,909 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/WALs/jenkins-hbase4.apache.org,40995,1682229223026/jenkins-hbase4.apache.org%2C40995%2C1682229223026.meta.1682229225887.meta 2023-04-23 05:53:45,909 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39671,DS-75cf324d-79ef-45be-8e7d-e86b031df2fc,DISK]] 2023-04-23 05:53:45,909 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:53:45,912 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-23 05:53:45,928 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-23 05:53:45,933 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-23 05:53:45,938 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-23 05:53:45,939 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:53:45,939 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-23 05:53:45,939 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-23 05:53:45,942 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-23 05:53:45,944 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/info 2023-04-23 05:53:45,944 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/info 2023-04-23 05:53:45,944 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-23 05:53:45,945 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:45,945 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-23 05:53:45,947 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/rep_barrier 2023-04-23 05:53:45,947 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/rep_barrier 2023-04-23 05:53:45,948 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-23 05:53:45,949 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:45,949 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-23 05:53:45,951 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/table 2023-04-23 05:53:45,951 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/table 2023-04-23 05:53:45,952 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-23 05:53:45,953 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:45,955 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740 2023-04-23 05:53:45,958 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740 2023-04-23 05:53:45,962 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:53:45,965 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-23 05:53:45,967 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59565622, jitterRate=-0.11240306496620178}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:53:45,967 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-23 05:53:45,978 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682229225857 2023-04-23 05:53:45,999 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-23 05:53:46,000 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-23 05:53:46,000 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,40995,1682229223026, state=OPEN 2023-04-23 05:53:46,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-23 05:53:46,004 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-23 05:53:46,009 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-23 05:53:46,009 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,40995,1682229223026 in 324 msec 2023-04-23 05:53:46,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-23 05:53:46,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 514 msec 2023-04-23 05:53:46,021 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5690 sec 2023-04-23 05:53:46,022 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682229226021, completionTime=-1 2023-04-23 05:53:46,022 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-23 05:53:46,023 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-23 05:53:46,095 DEBUG [hconnection-0x640c037f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:46,099 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56552, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:46,123 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-23 05:53:46,123 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682229286123 2023-04-23 05:53:46,123 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682229346123 2023-04-23 05:53:46,123 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 100 msec 2023-04-23 05:53:46,158 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44901,1682229221800-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:46,159 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44901,1682229221800-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:46,159 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44901,1682229221800-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:46,161 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:44901, period=300000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:46,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-23 05:53:46,168 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-23 05:53:46,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-23 05:53:46,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-23 05:53:46,188 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-23 05:53:46,191 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-23 05:53:46,194 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-23 05:53:46,215 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:53:46,218 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6 empty. 2023-04-23 05:53:46,219 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:53:46,219 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-23 05:53:46,642 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-23 05:53:46,645 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4c72146aba32a511e3acc1eb0b7832e6, NAME => 'hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp 2023-04-23 05:53:47,059 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:53:47,059 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 4c72146aba32a511e3acc1eb0b7832e6, disabling compactions & flushes 2023-04-23 05:53:47,060 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:53:47,060 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:53:47,060 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. after waiting 0 ms 2023-04-23 05:53:47,060 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:53:47,060 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:53:47,060 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 4c72146aba32a511e3acc1eb0b7832e6: 2023-04-23 05:53:47,065 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-23 05:53:47,085 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682229227069"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229227069"}]},"ts":"1682229227069"} 2023-04-23 05:53:47,114 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-23 05:53:47,117 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-23 05:53:47,122 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229227117"}]},"ts":"1682229227117"} 2023-04-23 05:53:47,127 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-23 05:53:47,139 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4c72146aba32a511e3acc1eb0b7832e6, ASSIGN}] 2023-04-23 05:53:47,142 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4c72146aba32a511e3acc1eb0b7832e6, ASSIGN 2023-04-23 05:53:47,145 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=4c72146aba32a511e3acc1eb0b7832e6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=false 2023-04-23 05:53:47,296 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4c72146aba32a511e3acc1eb0b7832e6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:47,296 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682229227295"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229227295"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229227295"}]},"ts":"1682229227295"} 2023-04-23 05:53:47,303 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 4c72146aba32a511e3acc1eb0b7832e6, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:53:47,464 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:53:47,465 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4c72146aba32a511e3acc1eb0b7832e6, NAME => 'hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:53:47,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:53:47,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:53:47,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:53:47,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:53:47,469 INFO [StoreOpener-4c72146aba32a511e3acc1eb0b7832e6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:53:47,472 DEBUG [StoreOpener-4c72146aba32a511e3acc1eb0b7832e6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6/info 2023-04-23 05:53:47,472 DEBUG [StoreOpener-4c72146aba32a511e3acc1eb0b7832e6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6/info 2023-04-23 05:53:47,472 INFO [StoreOpener-4c72146aba32a511e3acc1eb0b7832e6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4c72146aba32a511e3acc1eb0b7832e6 columnFamilyName info 2023-04-23 05:53:47,473 INFO [StoreOpener-4c72146aba32a511e3acc1eb0b7832e6-1] regionserver.HStore(310): Store=4c72146aba32a511e3acc1eb0b7832e6/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:47,475 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:53:47,476 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:53:47,481 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:53:47,485 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-23 05:53:47,486 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4c72146aba32a511e3acc1eb0b7832e6; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72418812, jitterRate=0.07912439107894897}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-23 05:53:47,486 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4c72146aba32a511e3acc1eb0b7832e6: 2023-04-23 05:53:47,488 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6., pid=6, masterSystemTime=1682229227457 2023-04-23 05:53:47,492 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:53:47,492 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:53:47,493 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4c72146aba32a511e3acc1eb0b7832e6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:47,494 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682229227493"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229227493"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229227493"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229227493"}]},"ts":"1682229227493"} 2023-04-23 05:53:47,501 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-23 05:53:47,501 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 4c72146aba32a511e3acc1eb0b7832e6, server=jenkins-hbase4.apache.org,40995,1682229223026 in 194 msec 2023-04-23 05:53:47,504 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-23 05:53:47,505 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=4c72146aba32a511e3acc1eb0b7832e6, ASSIGN in 363 msec 2023-04-23 05:53:47,506 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-23 05:53:47,507 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229227507"}]},"ts":"1682229227507"} 2023-04-23 05:53:47,510 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-23 05:53:47,513 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-23 05:53:47,516 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3340 sec 2023-04-23 05:53:47,592 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-23 05:53:47,594 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-23 05:53:47,594 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:53:47,631 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-23 05:53:47,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-23 05:53:47,662 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-04-23 05:53:47,666 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-23 05:53:47,680 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-23 05:53:47,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 19 msec 2023-04-23 05:53:47,699 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-23 05:53:47,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-23 05:53:47,702 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.598sec 2023-04-23 05:53:47,704 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-23 05:53:47,706 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-23 05:53:47,706 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-23 05:53:47,707 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44901,1682229221800-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-23 05:53:47,708 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44901,1682229221800-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-23 05:53:47,720 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-23 05:53:47,777 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x164139d4 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:47,783 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45b2cf6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:47,799 DEBUG [hconnection-0x1d2c4aea-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:47,811 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:47,824 INFO [Listener at localhost/41845] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:53:47,848 INFO [Listener at localhost/41845] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=208, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=373, ProcessCount=170, AvailableMemoryMB=13118 2023-04-23 05:53:47,866 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-23 05:53:47,871 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50852, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-23 05:53:47,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-23 05:53:47,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:53:47,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-23 05:53:47,892 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-23 05:53:47,894 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-23 05:53:47,898 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:47,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-23 05:53:47,899 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a empty. 2023-04-23 05:53:47,901 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:47,902 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:53:47,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-23 05:53:48,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-23 05:53:48,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-23 05:53:48,321 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-23 05:53:48,324 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => bfd1e7528fb104543b2ee708b991f02a, NAME => 'TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp 2023-04-23 05:53:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-23 05:53:48,738 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:53:48,738 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing bfd1e7528fb104543b2ee708b991f02a, disabling compactions & flushes 2023-04-23 05:53:48,738 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:48,738 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:48,738 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. after waiting 0 ms 2023-04-23 05:53:48,738 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:48,738 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:48,738 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:48,743 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-23 05:53:48,745 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229228744"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229228744"}]},"ts":"1682229228744"} 2023-04-23 05:53:48,747 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-23 05:53:48,749 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-23 05:53:48,749 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229228749"}]},"ts":"1682229228749"} 2023-04-23 05:53:48,751 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-23 05:53:48,755 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfd1e7528fb104543b2ee708b991f02a, ASSIGN}] 2023-04-23 05:53:48,757 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfd1e7528fb104543b2ee708b991f02a, ASSIGN 2023-04-23 05:53:48,758 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfd1e7528fb104543b2ee708b991f02a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=false 2023-04-23 05:53:48,910 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=bfd1e7528fb104543b2ee708b991f02a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:48,910 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229228910"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229228910"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229228910"}]},"ts":"1682229228910"} 2023-04-23 05:53:48,915 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:53:49,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-23 05:53:49,083 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:49,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bfd1e7528fb104543b2ee708b991f02a, NAME => 'TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:53:49,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:49,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:53:49,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:49,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:49,095 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:49,102 DEBUG [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A 2023-04-23 05:53:49,102 DEBUG [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A 2023-04-23 05:53:49,103 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:53:49,104 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bfd1e7528fb104543b2ee708b991f02a columnFamilyName A 2023-04-23 05:53:49,105 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] regionserver.HStore(310): Store=bfd1e7528fb104543b2ee708b991f02a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:49,105 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:49,111 DEBUG [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B 2023-04-23 05:53:49,111 DEBUG [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B 2023-04-23 05:53:49,111 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:53:49,113 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bfd1e7528fb104543b2ee708b991f02a columnFamilyName B 2023-04-23 05:53:49,114 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] regionserver.HStore(310): Store=bfd1e7528fb104543b2ee708b991f02a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:49,114 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:49,118 DEBUG [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C 2023-04-23 05:53:49,118 DEBUG [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C 2023-04-23 05:53:49,119 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:53:49,119 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bfd1e7528fb104543b2ee708b991f02a columnFamilyName C 2023-04-23 05:53:49,120 INFO [StoreOpener-bfd1e7528fb104543b2ee708b991f02a-1] regionserver.HStore(310): Store=bfd1e7528fb104543b2ee708b991f02a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:53:49,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:49,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:49,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:49,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:53:49,131 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:49,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-23 05:53:49,138 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bfd1e7528fb104543b2ee708b991f02a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69215018, jitterRate=0.03138414025306702}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:53:49,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:49,140 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., pid=11, masterSystemTime=1682229229073 2023-04-23 05:53:49,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:49,144 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:49,145 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=bfd1e7528fb104543b2ee708b991f02a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:49,145 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229229144"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229229144"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229229144"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229229144"}]},"ts":"1682229229144"} 2023-04-23 05:53:49,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-23 05:53:49,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 in 233 msec 2023-04-23 05:53:49,155 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-23 05:53:49,156 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfd1e7528fb104543b2ee708b991f02a, ASSIGN in 396 msec 2023-04-23 05:53:49,157 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-23 05:53:49,157 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229229157"}]},"ts":"1682229229157"} 2023-04-23 05:53:49,160 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-23 05:53:49,164 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-23 05:53:49,168 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2770 sec 2023-04-23 05:53:50,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-23 05:53:50,027 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-23 05:53:50,033 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x23162110 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,039 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39bf63a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,042 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,047 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56574, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,051 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-23 05:53:50,056 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50866, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-23 05:53:50,066 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x1e82daee to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,071 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@377933e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,071 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x36dc7f7f to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,076 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@401974c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,076 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x2e2005d0 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,081 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ca4a062, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,082 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x79fee867 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,087 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ec07124, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,088 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x75f5a5e6 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,095 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ef1151, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,096 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x64804459 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,102 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ae6c42b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,102 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x142b9819 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,128 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@726fb449, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,129 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbe28d to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,134 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6089dec1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,135 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x3c9ded19 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:53:50,139 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c72af8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:53:50,147 DEBUG [hconnection-0x1a5dd2ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,148 DEBUG [hconnection-0x5dfec272-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,152 DEBUG [hconnection-0x3df3bc44-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,156 DEBUG [hconnection-0x7e007515-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,161 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56580, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,162 DEBUG [hconnection-0x481e6d68-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,161 DEBUG [hconnection-0x5ab73287-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,162 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56582, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,159 DEBUG [hconnection-0x3685aa1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,159 DEBUG [hconnection-0x5de3642b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,163 DEBUG [hconnection-0x224073f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:53:50,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:50,175 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56592, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,175 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56606, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,177 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,177 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56614, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,178 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,178 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56638, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,191 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56652, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:53:50,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:50,201 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:50,201 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:50,208 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:50,208 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:50,212 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,212 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,214 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:50,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:50,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,224 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,225 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:50,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:50,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:50,252 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:50,254 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:50,267 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:53:50,268 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:50,256 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:50,275 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:50,276 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:50,277 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:50,287 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:50,297 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:50,297 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:50,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:50,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:50,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:50,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:50,298 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:50,310 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:50,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26781c23[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:50,310 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,317 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:50,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,318 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:50,318 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:50,318 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:50,331 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:50,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:50,339 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:50,339 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:50,339 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:50,339 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:50,339 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:50,359 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,359 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:50,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:50,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,390 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,390 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,390 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,390 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,390 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:50,390 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:50,390 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:50,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:50,390 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:50,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:50,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:50,396 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:50,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:50,399 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:50,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229290439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229290445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,499 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:50,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:50,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229290451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56652 deadline: 1682229290459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:50,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1682229290460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:50,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:50,564 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:50,564 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:50,578 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:50,578 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:50,585 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,585 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,587 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,587 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:50,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:50,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/4aaa4a03c43a4c2da3e6a21cb6ede65c 2023-04-23 05:53:50,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:50,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:50,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229290600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229290607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:50,612 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:50,612 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:50,613 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:50,616 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:50,616 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:50,616 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:50,616 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:50,621 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:50,621 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,621 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:50,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c5ead7c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:50,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:50,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:50,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:50,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:50,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229290630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,634 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:50,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56652 deadline: 1682229290632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:50,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1682229290636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:50,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:50,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:50,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:50,649 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:50,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:50,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:50,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:50,649 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:50,657 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,657 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,657 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:50,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:50,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:50,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,663 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:50,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:50,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:50,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:50,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,674 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:50,675 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:50,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:50,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:50,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:50,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:50,738 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-23 05:53:50,742 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-23 05:53:50,745 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-23 05:53:50,775 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:50,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:50,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:50,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229290807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229290813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:50,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:50,820 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:50,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:50,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:50,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:50,834 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,834 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,835 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:50,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,837 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:50,839 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,839 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:50,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:50,847 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:50,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:50,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229290841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,852 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:50,852 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:50,852 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:50,852 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:50,865 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,865 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:50,866 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,866 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56652 deadline: 1682229290859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:50,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:50,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:50,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1682229290866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:50,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11e36350[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:50,871 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:50,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:50,874 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,874 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:50,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,877 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:50,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:50,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:50,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:50,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:50,896 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:50,896 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:50,896 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:50,896 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:50,896 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:50,899 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,899 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,899 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:50,901 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,903 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:50,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,914 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:50,914 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:50,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:50,914 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:50,915 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:50,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:50,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:50,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:50,916 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:50,916 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:50,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:50,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:50,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:50,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,017 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:51,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:51,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:51,039 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:51,040 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,040 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:51,041 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:51,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:51,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:51,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:51,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:51,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:51,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:51,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:51,061 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:51,061 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:51,061 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:51,061 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:51,069 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:51,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:51,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29acf206[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:51,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:51,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:51,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:51,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:51,138 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,138 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:51,138 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:51,138 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229291140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229291140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:51,154 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:51,154 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:51,154 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:51,154 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:51,154 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:51,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:51,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229291156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,164 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,168 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/fdb8e3e24bf74e67b0fb1df448b84fcf 2023-04-23 05:53:51,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1682229291181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56652 deadline: 1682229291187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,201 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:51,201 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:51,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:51,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:51,206 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,206 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,207 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:51,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:51,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:51,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,208 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,208 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,208 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:51,208 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:51,308 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:51,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:51,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:51,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:51,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:51,319 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:51,319 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,320 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:51,320 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:51,320 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,320 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,322 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,322 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:51,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:51,323 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,323 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:51,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:51,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:51,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:51,324 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:51,324 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,325 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:51,325 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:51,326 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:51,326 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:51,326 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:51,326 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:51,326 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:51,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,330 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,330 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f537a14[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:51,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:51,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,335 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:51,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:51,340 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:51,341 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:51,350 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:51,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:51,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:51,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:51,351 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:51,353 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,353 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:51,353 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:51,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,365 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:51,365 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:51,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:51,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:51,366 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:51,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:51,366 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:51,366 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,366 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:51,366 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:51,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:51,366 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:51,467 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:51,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:51,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:51,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:51,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:51,477 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:51,477 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,478 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:51,478 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:51,479 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,479 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,482 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:51,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:51,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:51,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:51,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:51,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:51,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:51,498 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:51,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:51,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@260f1bcf[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:51,510 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:51,510 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,512 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:51,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:51,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:51,512 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,512 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:51,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:51,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:51,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:51,526 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:51,526 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:51,526 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:51,526 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:51,526 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:51,529 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,529 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:51,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:51,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,531 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,538 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,538 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,538 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:51,538 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:51,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:51,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:51,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:51,538 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:51,538 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:51,538 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,539 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,539 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,539 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:51,539 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:51,539 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:51,575 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-23 05:53:51,639 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:51,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:51,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:51,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229291649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229291649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:51,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:51,658 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:51,658 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,659 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:51,659 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:51,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,661 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:51,661 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,662 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,662 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:51,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:51,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:51,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:51,664 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:51,665 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:51,665 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:51,665 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:51,666 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:51,666 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:51,667 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:51,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229291667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,671 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,671 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c7c15bd[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:51,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:51,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:51,676 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,676 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:51,678 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:51,684 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:51,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:51,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:51,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:51,684 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:51,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:51,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:51,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:51,687 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1682229291686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,689 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:51,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/a49050e3142c4a1c816f48361793bac8 2023-04-23 05:53:51,697 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:51,697 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:51,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:51,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:51,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:51,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:51,700 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:51,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:51,700 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:51,700 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:51,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:51,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:51,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56652 deadline: 1682229291702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,709 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/4aaa4a03c43a4c2da3e6a21cb6ede65c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4aaa4a03c43a4c2da3e6a21cb6ede65c 2023-04-23 05:53:51,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4aaa4a03c43a4c2da3e6a21cb6ede65c, entries=250, sequenceid=14, filesize=16.2 K 2023-04-23 05:53:51,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/fdb8e3e24bf74e67b0fb1df448b84fcf as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/fdb8e3e24bf74e67b0fb1df448b84fcf 2023-04-23 05:53:51,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/fdb8e3e24bf74e67b0fb1df448b84fcf, entries=150, sequenceid=14, filesize=11.6 K 2023-04-23 05:53:51,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/a49050e3142c4a1c816f48361793bac8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a49050e3142c4a1c816f48361793bac8 2023-04-23 05:53:51,794 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a49050e3142c4a1c816f48361793bac8, entries=150, sequenceid=14, filesize=11.6 K 2023-04-23 05:53:51,798 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for bfd1e7528fb104543b2ee708b991f02a in 1531ms, sequenceid=14, compaction requested=false 2023-04-23 05:53:51,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:51,800 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:51,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:51,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:51,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:51,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:51,812 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:51,812 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,817 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:51,817 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:51,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:51,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,829 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:51,830 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:51,830 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:51,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:51,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:51,836 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:51,839 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:51,839 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:51,839 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:53:51,840 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:51,840 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:51,840 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:51,840 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:51,840 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:51,840 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:51,975 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-23 05:53:51,976 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-23 05:53:52,300 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a3d79f23b0a6421f9c71501906fe2892 2023-04-23 05:53:52,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:52,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:52,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:52,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229292695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:52,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1682229292697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:52,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229292698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:52,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229292698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:52,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56652 deadline: 1682229292712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,776 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/27eb1a047c5f46698ef42436f69b4411 2023-04-23 05:53:52,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:52,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229292802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:52,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229292811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:52,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229292811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,856 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/d96be4f3255c497cbee34c3023ee22e3 2023-04-23 05:53:52,873 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a3d79f23b0a6421f9c71501906fe2892 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3d79f23b0a6421f9c71501906fe2892 2023-04-23 05:53:52,885 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3d79f23b0a6421f9c71501906fe2892, entries=150, sequenceid=34, filesize=11.6 K 2023-04-23 05:53:52,889 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/27eb1a047c5f46698ef42436f69b4411 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/27eb1a047c5f46698ef42436f69b4411 2023-04-23 05:53:52,910 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/27eb1a047c5f46698ef42436f69b4411, entries=150, sequenceid=34, filesize=11.6 K 2023-04-23 05:53:52,913 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/d96be4f3255c497cbee34c3023ee22e3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d96be4f3255c497cbee34c3023ee22e3 2023-04-23 05:53:52,957 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d96be4f3255c497cbee34c3023ee22e3, entries=150, sequenceid=34, filesize=11.6 K 2023-04-23 05:53:52,960 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for bfd1e7528fb104543b2ee708b991f02a in 1120ms, sequenceid=34, compaction requested=false 2023-04-23 05:53:52,960 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:52,960 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:52,960 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:52,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:52,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:52,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:52,963 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:52,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:52,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:52,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@628eeef6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:52,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:52,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:52,969 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:52,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:52,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:52,973 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:52,973 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:52,973 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:52,974 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:52,980 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:52,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:52,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:52,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:52,980 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:52,985 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:52,985 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:52,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:52,986 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:52,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:52,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:52,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,992 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:52,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:52,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:52,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:52,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:52,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:52,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:52,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:52,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:52,999 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:52,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:53,001 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:53,001 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:53,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:53,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:53,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-23 05:53:53,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:53,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:53,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:53,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:53,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:53,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:53,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/3f478cad4bd442ce8fc7ca0ed0629c21 2023-04-23 05:53:53,102 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:53,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:53,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/7a4755c024234bb2831fd759d93dbee7 2023-04-23 05:53:53,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:53,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:53,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:53,119 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:53,119 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,123 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:53,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:53,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:53,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:53,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:53,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:53,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:53,133 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:53,133 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:53,136 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:53,136 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,136 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:53,137 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:53,137 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,141 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:53,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:53,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:53,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:53,145 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,145 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229293133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3659ab69[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:53,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:53,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229293148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229293148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:53,161 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:53,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:53,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:53,175 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:53,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:53,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:53,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:53,175 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:53,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/149003e3dc6c4b2993336f2dbeb9d249 2023-04-23 05:53:53,183 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,183 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:53,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,184 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,196 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/3f478cad4bd442ce8fc7ca0ed0629c21 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3f478cad4bd442ce8fc7ca0ed0629c21 2023-04-23 05:53:53,205 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,205 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,205 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,205 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,205 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,205 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,205 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:53,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:53,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:53,205 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:53,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:53,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:53,216 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:53,216 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:53,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3f478cad4bd442ce8fc7ca0ed0629c21, entries=200, sequenceid=48, filesize=13.9 K 2023-04-23 05:53:53,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/7a4755c024234bb2831fd759d93dbee7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/7a4755c024234bb2831fd759d93dbee7 2023-04-23 05:53:53,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/7a4755c024234bb2831fd759d93dbee7, entries=150, sequenceid=48, filesize=11.6 K 2023-04-23 05:53:53,241 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/149003e3dc6c4b2993336f2dbeb9d249 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/149003e3dc6c4b2993336f2dbeb9d249 2023-04-23 05:53:53,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229293249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/149003e3dc6c4b2993336f2dbeb9d249, entries=150, sequenceid=48, filesize=11.6 K 2023-04-23 05:53:53,260 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for bfd1e7528fb104543b2ee708b991f02a in 247ms, sequenceid=48, compaction requested=true 2023-04-23 05:53:53,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:53,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:53,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:53:53,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:53:53,264 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:53,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:53,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:53,266 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:53:53,272 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:53:53,274 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:53:53,275 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/A is initiating minor compaction (all files) 2023-04-23 05:53:53,275 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/A in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,276 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4aaa4a03c43a4c2da3e6a21cb6ede65c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3d79f23b0a6421f9c71501906fe2892, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3f478cad4bd442ce8fc7ca0ed0629c21] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=41.7 K 2023-04-23 05:53:53,280 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/B is initiating minor compaction (all files) 2023-04-23 05:53:53,280 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/B in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,281 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/fdb8e3e24bf74e67b0fb1df448b84fcf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/27eb1a047c5f46698ef42436f69b4411, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/7a4755c024234bb2831fd759d93dbee7] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=34.8 K 2023-04-23 05:53:53,281 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 4aaa4a03c43a4c2da3e6a21cb6ede65c, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682229230208 2023-04-23 05:53:53,283 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting fdb8e3e24bf74e67b0fb1df448b84fcf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682229230227 2023-04-23 05:53:53,284 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 27eb1a047c5f46698ef42436f69b4411, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682229230438 2023-04-23 05:53:53,285 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a3d79f23b0a6421f9c71501906fe2892, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682229230438 2023-04-23 05:53:53,285 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7a4755c024234bb2831fd759d93dbee7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229232690 2023-04-23 05:53:53,286 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 3f478cad4bd442ce8fc7ca0ed0629c21, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229232688 2023-04-23 05:53:53,320 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:53,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:53,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:53,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/0160d4b395cd41368de3e39389e2b9dd 2023-04-23 05:53:53,333 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#B#compaction#10 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:53:53,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229293321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229293336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:53,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:53,349 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:53,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:53,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:53,353 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,353 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:53,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:53,368 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#A#compaction#12 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:53:53,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:53,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:53,383 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:53,383 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:53,383 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:53,383 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:53,384 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:53,384 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,384 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:53,384 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:53,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:53,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68560f0e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:53,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:53,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,423 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/d959a4583d374507bb415b28ddf8d829 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/d959a4583d374507bb415b28ddf8d829 2023-04-23 05:53:53,424 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:53,424 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,424 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:53,427 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:53,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,428 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,434 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:53,455 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:53,455 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:53,455 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:53,455 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:53,455 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:53,458 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229293451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,459 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,467 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229293453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229293456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,473 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/B of bfd1e7528fb104543b2ee708b991f02a into d959a4583d374507bb415b28ddf8d829(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:53:53,473 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:53,473 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/B, priority=13, startTime=1682229233263; duration=0sec 2023-04-23 05:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,474 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:53:53,474 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:53:53,474 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,481 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/f191e8d6804f4bf793bffddb08053351 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/f191e8d6804f4bf793bffddb08053351 2023-04-23 05:53:53,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:53,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:53,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:53,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:53,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:53,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:53,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:53,483 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:53,484 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:53,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:53,484 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:53:53,484 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/C is initiating minor compaction (all files) 2023-04-23 05:53:53,484 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/C in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,484 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a49050e3142c4a1c816f48361793bac8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d96be4f3255c497cbee34c3023ee22e3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/149003e3dc6c4b2993336f2dbeb9d249] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=34.8 K 2023-04-23 05:53:53,485 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a49050e3142c4a1c816f48361793bac8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682229230227 2023-04-23 05:53:53,486 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d96be4f3255c497cbee34c3023ee22e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682229230438 2023-04-23 05:53:53,486 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 149003e3dc6c4b2993336f2dbeb9d249, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229232690 2023-04-23 05:53:53,505 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/A of bfd1e7528fb104543b2ee708b991f02a into f191e8d6804f4bf793bffddb08053351(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:53:53,509 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#C#compaction#13 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:53:53,513 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:53,513 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/A, priority=13, startTime=1682229233263; duration=0sec 2023-04-23 05:53:53,514 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:53:53,567 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/ff0e9490797b483aa9231e9c9e558dc6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ff0e9490797b483aa9231e9c9e558dc6 2023-04-23 05:53:53,579 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/C of bfd1e7528fb104543b2ee708b991f02a into ff0e9490797b483aa9231e9c9e558dc6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:53:53,582 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:53,583 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/C, priority=13, startTime=1682229233264; duration=0sec 2023-04-23 05:53:53,583 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:53:53,584 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:53,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:53,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:53,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:53,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:53,600 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:53,600 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,601 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:53,601 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:53,602 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,602 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,603 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:53,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:53,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:53,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:53,613 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:53,613 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:53,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:53,618 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,618 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:53,618 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:53,619 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:53,623 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,623 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244b2e44[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:53,627 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:53,627 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:53,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:53,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:53,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:53,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:53,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:53,636 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:53,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:53,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:53,637 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:53,637 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:53,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:53,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,643 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:53,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:53,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,651 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:53,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:53,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:53,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:53,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:53,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,653 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,653 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,653 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:53,653 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:53,653 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,653 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:53,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229293670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229293671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,753 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:53,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:53,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:53,764 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:53,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,765 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:53,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:53,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:53,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:53,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:53,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:53,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:53,771 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:53,771 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:53,776 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:53,776 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:53,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229293779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,781 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,781 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:53,782 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:53,782 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:53,782 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,787 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:53,787 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31279d65[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:53,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:53,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:53,793 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,793 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:53,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:53,795 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:53,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:53,805 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:53,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:53,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:53,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:53,805 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:53,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:53,809 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,811 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,817 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:53,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:53,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:53,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:53,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:53,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:53,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:53,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,821 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:53,821 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:53,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,822 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:53,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/b0ea5bedc5e24942b78b12e3615bf163 2023-04-23 05:53:53,922 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:53,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:53,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=68 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/5809d075b4f642038cd20d0cff85c082 2023-04-23 05:53:53,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:53,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:53,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:53,937 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:53,937 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,940 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:53,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:53,942 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,942 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,944 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:53,944 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:53,949 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,949 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:53,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/0160d4b395cd41368de3e39389e2b9dd as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0160d4b395cd41368de3e39389e2b9dd 2023-04-23 05:53:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:53,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:53,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0160d4b395cd41368de3e39389e2b9dd, entries=200, sequenceid=68, filesize=13.9 K 2023-04-23 05:53:53,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:53,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/b0ea5bedc5e24942b78b12e3615bf163 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b0ea5bedc5e24942b78b12e3615bf163 2023-04-23 05:53:53,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:53,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:53,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:53,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:53,972 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,972 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:53,972 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:53,972 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:53,973 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:53,973 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:53,973 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,973 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:53,973 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,982 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229293983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:53,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229293983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5e7acb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:53,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:53,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:53,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:53,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:53,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:53,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:53,993 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:53,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b0ea5bedc5e24942b78b12e3615bf163, entries=150, sequenceid=68, filesize=11.6 K 2023-04-23 05:53:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:53,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/5809d075b4f642038cd20d0cff85c082 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5809d075b4f642038cd20d0cff85c082 2023-04-23 05:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:54,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:54,000 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:54,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:54,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:54,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:54,000 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:54,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:54,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:54,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:54,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,004 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,008 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5809d075b4f642038cd20d0cff85c082, entries=150, sequenceid=68, filesize=11.6 K 2023-04-23 05:53:54,016 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:54,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bfd1e7528fb104543b2ee708b991f02a in 752ms, sequenceid=68, compaction requested=false 2023-04-23 05:53:54,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:54,017 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:54,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:54,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:54,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:54,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:54,027 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:54,027 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:54,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,028 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,028 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,028 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:54,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:54,128 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:54,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:54,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:54,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:54,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:54,136 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:54,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:54,137 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:54,137 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:54,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:54,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:54,141 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,141 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:54,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:54,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:54,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:54,145 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:54,145 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:53:54,145 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:54,145 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:54,146 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:54,146 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:54,146 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:54,146 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:54,195 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/0a8b75b0b7624aa1919f8a120e19192e 2023-04-23 05:53:54,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:54,290 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:54,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:54,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229294373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229294477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229294490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229294491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,628 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/22bddd084cb24915808eae9ae041e44f 2023-04-23 05:53:54,674 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/c7439bec5fc04786a7f48781df577aa6 2023-04-23 05:53:54,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:54,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229294681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,688 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/0a8b75b0b7624aa1919f8a120e19192e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0a8b75b0b7624aa1919f8a120e19192e 2023-04-23 05:53:54,700 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0a8b75b0b7624aa1919f8a120e19192e, entries=150, sequenceid=83, filesize=11.6 K 2023-04-23 05:53:54,704 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/22bddd084cb24915808eae9ae041e44f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/22bddd084cb24915808eae9ae041e44f 2023-04-23 05:53:54,713 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/22bddd084cb24915808eae9ae041e44f, entries=150, sequenceid=83, filesize=11.6 K 2023-04-23 05:53:54,719 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/c7439bec5fc04786a7f48781df577aa6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c7439bec5fc04786a7f48781df577aa6 2023-04-23 05:53:54,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:54,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1682229294719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,726 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4266 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:53:54,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:54,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56652 deadline: 1682229294728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,733 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4275 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:53:54,740 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c7439bec5fc04786a7f48781df577aa6, entries=150, sequenceid=83, filesize=11.6 K 2023-04-23 05:53:54,742 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bfd1e7528fb104543b2ee708b991f02a in 597ms, sequenceid=83, compaction requested=true 2023-04-23 05:53:54,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:54,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:54,742 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:54,743 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,748 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:54,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:54,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a822f1[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:54,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:54,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,753 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:54,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:54,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:54,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:54,757 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:54,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:54,766 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:54,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:54,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:54,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:54,766 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:54,769 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,770 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:54,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,770 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:54,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:54,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:54,772 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,774 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,781 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:54,781 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:54,783 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:54,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:54,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:54,783 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:54,783 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:54,781 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:54,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:54,783 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:54,783 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:54,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:54,883 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:54,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:54,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:54,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:54,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:54,890 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:54,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:54,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:54,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:54,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,893 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:54,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:54,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:54,894 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,894 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:54,894 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:54,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:54,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:54,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:54,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:54,896 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:54,896 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:54,897 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:54,900 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:54,900 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:54,901 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:53:54,901 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:54,901 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:54,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:54,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:54,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:54,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:54,946 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/2d982d00b5b24c6da99d8aae1069adff 2023-04-23 05:53:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:54,990 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:55,055 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/cebb8893d7ab4ca5930cafa06f84e027 2023-04-23 05:53:55,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229295066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,119 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/beaaddb75dde46518d79814f03a36d47 2023-04-23 05:53:55,130 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/2d982d00b5b24c6da99d8aae1069adff as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/2d982d00b5b24c6da99d8aae1069adff 2023-04-23 05:53:55,141 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/2d982d00b5b24c6da99d8aae1069adff, entries=150, sequenceid=103, filesize=11.6 K 2023-04-23 05:53:55,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/cebb8893d7ab4ca5930cafa06f84e027 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/cebb8893d7ab4ca5930cafa06f84e027 2023-04-23 05:53:55,164 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/cebb8893d7ab4ca5930cafa06f84e027, entries=150, sequenceid=103, filesize=11.6 K 2023-04-23 05:53:55,170 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/beaaddb75dde46518d79814f03a36d47 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/beaaddb75dde46518d79814f03a36d47 2023-04-23 05:53:55,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229295174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,190 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/beaaddb75dde46518d79814f03a36d47, entries=150, sequenceid=103, filesize=11.6 K 2023-04-23 05:53:55,195 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bfd1e7528fb104543b2ee708b991f02a in 294ms, sequenceid=103, compaction requested=true 2023-04-23 05:53:55,195 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:55,195 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:55,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:55,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:55,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:55,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:55,200 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:55,201 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,201 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6c07ee[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:55,208 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:55,208 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,222 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:55,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:55,226 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:55,232 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:55,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:55,239 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:55,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:55,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:55,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:55,239 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:55,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:55,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:55,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:55,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:55,252 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:55,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:55,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:55,252 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:55,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:55,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,257 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:55,258 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:55,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:55,358 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:55,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:55,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:55,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:55,364 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:55,364 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:55,365 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:55,365 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:55,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:55,368 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:55,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:55,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:55,369 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,369 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:55,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:55,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:55,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:55,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:55,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:55,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:55,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:55,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:55,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:55,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:55,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:55,372 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:53:55,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:55,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:55,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:55,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:55,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:55,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:55,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:55,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:55,424 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/932c18fb02644d07b406c1b743701eb7 2023-04-23 05:53:55,461 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/a16cbb31c1dc4496bf56c0978e4a77a1 2023-04-23 05:53:55,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229295506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229295508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229235476, max timestamp=1682229235507Num uniques 859; 2023-04-23 05:53:55,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229235476, max timestamp=1682229235507Num uniques 859; 2023-04-23 05:53:55,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:53:55,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:53:55,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229235476, max timestamp=1682229235507Num uniques 859; 2023-04-23 05:53:55,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:53:55,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229295615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229295616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229295619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229295725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229295818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229295829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,909 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/f4db1da8c66347bb8e1b7ee40052a805 2023-04-23 05:53:55,918 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/932c18fb02644d07b406c1b743701eb7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/932c18fb02644d07b406c1b743701eb7 2023-04-23 05:53:55,926 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/932c18fb02644d07b406c1b743701eb7, entries=150, sequenceid=115, filesize=11.6 K 2023-04-23 05:53:55,929 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/a16cbb31c1dc4496bf56c0978e4a77a1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a16cbb31c1dc4496bf56c0978e4a77a1 2023-04-23 05:53:55,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:55,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229295930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,938 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a16cbb31c1dc4496bf56c0978e4a77a1, entries=150, sequenceid=115, filesize=11.6 K 2023-04-23 05:53:55,941 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/f4db1da8c66347bb8e1b7ee40052a805 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/f4db1da8c66347bb8e1b7ee40052a805 2023-04-23 05:53:55,951 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/f4db1da8c66347bb8e1b7ee40052a805, entries=150, sequenceid=115, filesize=11.6 K 2023-04-23 05:53:55,953 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for bfd1e7528fb104543b2ee708b991f02a in 581ms, sequenceid=115, compaction requested=true 2023-04-23 05:53:55,953 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:55,954 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:55,954 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:55,957 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,957 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:55,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62ad73d3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:55,959 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:55,960 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,961 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:55,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:55,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:55,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:55,976 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:55,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:55,977 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:55,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:55,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:55,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:55,977 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:55,980 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,980 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:55,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,981 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:55,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:55,984 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:55,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:55,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:55,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:55,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:55,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:55,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:55,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:55,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:55,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:55,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:55,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:55,995 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:55,996 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:55,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:56,096 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:56,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:56,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:56,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:56,102 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:56,102 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:56,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:56,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:56,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:56,105 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:56,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:56,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:56,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:56,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:56,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:56,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:56,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:56,113 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=120.76 KB heapSize=345.14 KB 2023-04-23 05:53:56,114 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:56,114 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:53:56,114 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:56,114 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:53:56,114 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:56,115 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:53:56,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:56,133 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:56,152 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/8ed391f0c7ee492c94cbe5a2a804067b 2023-04-23 05:53:56,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229296168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229296168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,201 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/47f7f431ec9a4f24868a43b6aab7cea1 2023-04-23 05:53:56,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229296238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229296274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229296285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229296477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229296492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,633 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/a14e4fb37eb44109a342781c3ec49cb4 2023-04-23 05:53:56,641 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/8ed391f0c7ee492c94cbe5a2a804067b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/8ed391f0c7ee492c94cbe5a2a804067b 2023-04-23 05:53:56,648 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/8ed391f0c7ee492c94cbe5a2a804067b, entries=150, sequenceid=150, filesize=11.8 K 2023-04-23 05:53:56,651 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/47f7f431ec9a4f24868a43b6aab7cea1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/47f7f431ec9a4f24868a43b6aab7cea1 2023-04-23 05:53:56,660 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/47f7f431ec9a4f24868a43b6aab7cea1, entries=150, sequenceid=150, filesize=11.8 K 2023-04-23 05:53:56,663 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/a14e4fb37eb44109a342781c3ec49cb4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a14e4fb37eb44109a342781c3ec49cb4 2023-04-23 05:53:56,676 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a14e4fb37eb44109a342781c3ec49cb4, entries=150, sequenceid=150, filesize=11.8 K 2023-04-23 05:53:56,681 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=67.09 KB/68700 for bfd1e7528fb104543b2ee708b991f02a in 565ms, sequenceid=150, compaction requested=true 2023-04-23 05:53:56,681 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:56,681 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:56,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:56,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:56,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:56,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:56,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b9cd56f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:56,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:56,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,689 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,689 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,690 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:56,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:56,691 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:56,691 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:56,691 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,691 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:56,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:56,699 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:56,699 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:56,699 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:56,699 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:56,699 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:56,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:56,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:56,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:56,702 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,704 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:56,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:56,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:56,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,709 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:56,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:56,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:56,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,713 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:56,713 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:56,713 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:56,713 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:56,713 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:56,750 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-23 05:53:56,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:56,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:56,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:56,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:56,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:56,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:56,813 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:56,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:56,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229296811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:56,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:56,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:56,819 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:56,819 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:56,822 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:56,822 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:56,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229296814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229296815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,825 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:56,825 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:56,825 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,825 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:56,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:56,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:56,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:56,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:56,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:56,828 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:56,828 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:56,828 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:56,828 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:56,829 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:56,829 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:56,829 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,829 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:56,829 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:56,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:56,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:56,830 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:56,831 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,831 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:56,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:56,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8f0db47[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:56,834 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:56,834 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,836 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,836 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,836 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:56,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:56,841 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,841 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:56,841 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:56,841 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:56,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:56,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:56,846 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:56,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:56,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:56,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:56,846 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:56,851 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,851 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:56,851 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:56,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:56,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:56,854 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:56,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,861 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,864 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:56,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:56,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:56,864 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:56,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,864 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:56,865 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:56,865 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:56,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:56,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:56,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229296916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229296925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:56,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229296925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,965 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:56,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:56,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:56,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:56,972 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:56,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:56,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:56,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:56,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,978 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,979 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,979 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:56,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:56,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:56,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:56,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:56,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:56,988 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:56,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:56,992 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:56,992 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:56,993 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:56,993 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:56,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:56,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:56,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:56,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:57,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:57,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7acfec0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:57,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:57,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:57,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:57,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:57,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:57,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:57,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:57,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:57,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:57,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:57,014 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:57,014 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:57,014 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:57,014 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:57,014 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:57,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:57,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:57,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,019 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,019 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:57,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:57,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:57,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,025 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:57,025 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:57,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:57,025 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:57,025 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,025 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:57,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:57,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:57,025 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:57,027 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:57,027 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:57,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:57,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229297121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,127 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:57,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:57,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:57,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229297134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:57,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229297135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:57,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:57,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:57,152 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:57,153 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:57,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:57,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:57,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:57,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:57,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:57,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:57,161 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:57,161 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:57,164 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:57,164 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:57,164 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:57,165 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,167 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e96260d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:57,170 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:57,170 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,171 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:57,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:57,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:57,172 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,172 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:57,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/723e2b0c5c1a48c6bebed84398e7eb05 2023-04-23 05:53:57,174 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:57,174 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:57,174 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:57,176 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:57,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:57,181 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:57,181 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:57,181 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:57,182 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:57,182 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:57,184 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,184 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,184 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:57,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:57,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:57,186 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,188 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:57,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:57,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:57,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:57,192 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:57,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:57,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:57,198 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,200 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:57,200 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:57,300 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:57,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:57,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:57,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:57,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:57,305 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:57,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:57,306 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:57,306 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:57,306 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,306 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,337 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:57,337 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,338 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,338 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:57,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:57,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:57,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:57,344 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:57,344 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:57,344 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:57,345 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:57,345 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:57,345 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:57,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:57,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:57,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229297428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,437 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:57,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339cbd1d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:57,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:57,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:57,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229297440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:57,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229297443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:57,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:57,590 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,590 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:57,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:57,593 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:57,593 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:57,593 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:57,593 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:57,594 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:57,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/c09a077f111842b2b19fb2bc82c26c80 2023-04-23 05:53:57,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/ddddc13f5b5a40dfa4b3ce50f6d308c1 2023-04-23 05:53:57,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/723e2b0c5c1a48c6bebed84398e7eb05 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/723e2b0c5c1a48c6bebed84398e7eb05 2023-04-23 05:53:57,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/723e2b0c5c1a48c6bebed84398e7eb05, entries=200, sequenceid=164, filesize=14.1 K 2023-04-23 05:53:57,683 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,683 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,683 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:57,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/c09a077f111842b2b19fb2bc82c26c80 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c09a077f111842b2b19fb2bc82c26c80 2023-04-23 05:53:57,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:57,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:57,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:57,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c09a077f111842b2b19fb2bc82c26c80, entries=150, sequenceid=164, filesize=11.8 K 2023-04-23 05:53:57,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/ddddc13f5b5a40dfa4b3ce50f6d308c1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ddddc13f5b5a40dfa4b3ce50f6d308c1 2023-04-23 05:53:57,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ddddc13f5b5a40dfa4b3ce50f6d308c1, entries=150, sequenceid=164, filesize=11.8 K 2023-04-23 05:53:57,709 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for bfd1e7528fb104543b2ee708b991f02a in 959ms, sequenceid=164, compaction requested=true 2023-04-23 05:53:57,709 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:57,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:53:57,710 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-23 05:53:57,710 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-23 05:53:57,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:53:57,710 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:53:57,713 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88335 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-23 05:53:57,714 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/A is initiating minor compaction (all files) 2023-04-23 05:53:57,714 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/A in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:57,714 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/f191e8d6804f4bf793bffddb08053351, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0160d4b395cd41368de3e39389e2b9dd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0a8b75b0b7624aa1919f8a120e19192e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/2d982d00b5b24c6da99d8aae1069adff, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/932c18fb02644d07b406c1b743701eb7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/8ed391f0c7ee492c94cbe5a2a804067b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/723e2b0c5c1a48c6bebed84398e7eb05] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=86.3 K 2023-04-23 05:53:57,714 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-23 05:53:57,714 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/B is initiating minor compaction (all files) 2023-04-23 05:53:57,715 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/B in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:57,715 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/d959a4583d374507bb415b28ddf8d829, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b0ea5bedc5e24942b78b12e3615bf163, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/22bddd084cb24915808eae9ae041e44f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/cebb8893d7ab4ca5930cafa06f84e027, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a16cbb31c1dc4496bf56c0978e4a77a1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/47f7f431ec9a4f24868a43b6aab7cea1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c09a077f111842b2b19fb2bc82c26c80] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=81.6 K 2023-04-23 05:53:57,715 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting f191e8d6804f4bf793bffddb08053351, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229232690 2023-04-23 05:53:57,717 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d959a4583d374507bb415b28ddf8d829, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229232690 2023-04-23 05:53:57,717 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 0160d4b395cd41368de3e39389e2b9dd, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682229233130 2023-04-23 05:53:57,719 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b0ea5bedc5e24942b78b12e3615bf163, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682229233130 2023-04-23 05:53:57,719 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 22bddd084cb24915808eae9ae041e44f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682229233295 2023-04-23 05:53:57,720 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting cebb8893d7ab4ca5930cafa06f84e027, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1682229234354 2023-04-23 05:53:57,721 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 0a8b75b0b7624aa1919f8a120e19192e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682229233295 2023-04-23 05:53:57,721 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a16cbb31c1dc4496bf56c0978e4a77a1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682229235009 2023-04-23 05:53:57,721 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 2d982d00b5b24c6da99d8aae1069adff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1682229234354 2023-04-23 05:53:57,722 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 47f7f431ec9a4f24868a43b6aab7cea1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682229235592 2023-04-23 05:53:57,722 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 932c18fb02644d07b406c1b743701eb7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682229235009 2023-04-23 05:53:57,723 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 8ed391f0c7ee492c94cbe5a2a804067b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682229235592 2023-04-23 05:53:57,723 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c09a077f111842b2b19fb2bc82c26c80, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682229236157 2023-04-23 05:53:57,724 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 723e2b0c5c1a48c6bebed84398e7eb05, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682229236157 2023-04-23 05:53:57,804 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#B#compaction#31 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:53:57,807 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#A#compaction#30 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:53:57,834 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/aabd230b48464c2fa6a7b5b3554cbf4e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/aabd230b48464c2fa6a7b5b3554cbf4e 2023-04-23 05:53:57,849 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/B of bfd1e7528fb104543b2ee708b991f02a into aabd230b48464c2fa6a7b5b3554cbf4e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:53:57,849 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:57,849 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/B, priority=9, startTime=1682229237710; duration=0sec 2023-04-23 05:53:57,849 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:53:57,851 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-23 05:53:57,854 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83605 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-23 05:53:57,855 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/C is initiating minor compaction (all files) 2023-04-23 05:53:57,855 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/C in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:57,855 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ff0e9490797b483aa9231e9c9e558dc6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5809d075b4f642038cd20d0cff85c082, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c7439bec5fc04786a7f48781df577aa6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/beaaddb75dde46518d79814f03a36d47, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/f4db1da8c66347bb8e1b7ee40052a805, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a14e4fb37eb44109a342781c3ec49cb4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ddddc13f5b5a40dfa4b3ce50f6d308c1] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=81.6 K 2023-04-23 05:53:57,855 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ff0e9490797b483aa9231e9c9e558dc6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229232690 2023-04-23 05:53:57,856 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 5809d075b4f642038cd20d0cff85c082, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=68, earliestPutTs=1682229233130 2023-04-23 05:53:57,857 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c7439bec5fc04786a7f48781df577aa6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682229233295 2023-04-23 05:53:57,858 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting beaaddb75dde46518d79814f03a36d47, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1682229234354 2023-04-23 05:53:57,858 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f4db1da8c66347bb8e1b7ee40052a805, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1682229235009 2023-04-23 05:53:57,859 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a14e4fb37eb44109a342781c3ec49cb4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1682229235592 2023-04-23 05:53:57,860 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ddddc13f5b5a40dfa4b3ce50f6d308c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682229236157 2023-04-23 05:53:57,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:57,883 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:57,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:57,883 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:57,883 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:57,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:57,883 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:57,883 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,883 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,883 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:57,883 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,884 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:57,884 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:57,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:57,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:57,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:57,889 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#C#compaction#32 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:53:57,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:57,934 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:53:57,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:57,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:57,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:57,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:57,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:57,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:57,984 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:57,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:57,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:57,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229297980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:57,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229297984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:57,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:57,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229297985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:57,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:57,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:57,992 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:57,992 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:57,992 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:57,992 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:57,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:57,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:58,060 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:58,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:58,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:58,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:58,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:58,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:58,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:58,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:58,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:58,062 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:58,062 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:58,062 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:58,062 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:58,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:58,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229298087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229298090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229298091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:58,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:58,148 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:58,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:58,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3872ca[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:58,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:58,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:58,223 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:58,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:58,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:58,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:58,260 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/69a21fbb7bad401ab1b9d14a32a98313 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/69a21fbb7bad401ab1b9d14a32a98313 2023-04-23 05:53:58,269 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/A of bfd1e7528fb104543b2ee708b991f02a into 69a21fbb7bad401ab1b9d14a32a98313(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:53:58,269 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:58,269 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/A, priority=9, startTime=1682229237709; duration=0sec 2023-04-23 05:53:58,269 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:53:58,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,275 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,275 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:58,275 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:58,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:58,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:58,279 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:58,279 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:58,279 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:58,279 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:58,279 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:58,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229298291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56592 deadline: 1682229298297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229298297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,306 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/b412b4919e9f4f13b0d1aa27dd0a0f36 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/b412b4919e9f4f13b0d1aa27dd0a0f36 2023-04-23 05:53:58,325 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/C of bfd1e7528fb104543b2ee708b991f02a into b412b4919e9f4f13b0d1aa27dd0a0f36(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:53:58,325 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:58,325 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/C, priority=9, startTime=1682229237710; duration=0sec 2023-04-23 05:53:58,325 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:53:58,338 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,338 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:58,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,338 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:58,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:58,339 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,342 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/e7d210f892df4ee0815909adf9dca9ce 2023-04-23 05:53:58,382 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/35f032e7857a4548aefadde93266984f 2023-04-23 05:53:58,399 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:58,399 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:58,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:58,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:58,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:58,399 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:58,400 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:58,400 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:58,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:58,404 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,404 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:58,404 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:58,404 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,404 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:58,404 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:58,404 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/a1919cdbc76e4028ab7bd6538902627f 2023-04-23 05:53:58,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/e7d210f892df4ee0815909adf9dca9ce as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/e7d210f892df4ee0815909adf9dca9ce 2023-04-23 05:53:58,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/e7d210f892df4ee0815909adf9dca9ce, entries=200, sequenceid=185, filesize=14.1 K 2023-04-23 05:53:58,437 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/35f032e7857a4548aefadde93266984f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/35f032e7857a4548aefadde93266984f 2023-04-23 05:53:58,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/35f032e7857a4548aefadde93266984f, entries=150, sequenceid=185, filesize=11.8 K 2023-04-23 05:53:58,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/a1919cdbc76e4028ab7bd6538902627f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a1919cdbc76e4028ab7bd6538902627f 2023-04-23 05:53:58,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a1919cdbc76e4028ab7bd6538902627f, entries=150, sequenceid=185, filesize=11.8 K 2023-04-23 05:53:58,470 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bfd1e7528fb104543b2ee708b991f02a in 536ms, sequenceid=185, compaction requested=false 2023-04-23 05:53:58,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:58,500 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:58,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:58,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:58,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:58,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:58,507 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:58,507 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:58,507 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:58,507 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:58,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:58,529 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,529 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:58,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:58,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:58,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:58,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:58,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:58,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:58,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:58,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:58,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:58,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:58,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:58,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:58,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:58,541 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:58,541 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:58,541 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:53:58,541 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:58,541 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:58,542 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:58,542 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:58,542 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:58,542 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:58,585 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/5f7938b5ac3f47c99df255d9f07269f0 2023-04-23 05:53:58,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:58,607 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:58,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229298630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229298632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229298634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229298736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229298736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229298738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1682229298767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,769 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8309 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:53:58,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56652 deadline: 1682229298770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,778 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8319 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:53:58,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229298942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229298943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:58,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:58,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229298944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,027 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/dc104c27820c45f783ea087c5b52887d 2023-04-23 05:53:59,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:59,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229299245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:59,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229299249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:59,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229299249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,450 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/aabcf50ab17a4896ad360bd4f358dbf2 2023-04-23 05:53:59,461 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/5f7938b5ac3f47c99df255d9f07269f0 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5f7938b5ac3f47c99df255d9f07269f0 2023-04-23 05:53:59,469 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5f7938b5ac3f47c99df255d9f07269f0, entries=150, sequenceid=199, filesize=11.8 K 2023-04-23 05:53:59,471 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/dc104c27820c45f783ea087c5b52887d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/dc104c27820c45f783ea087c5b52887d 2023-04-23 05:53:59,480 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/dc104c27820c45f783ea087c5b52887d, entries=150, sequenceid=199, filesize=11.8 K 2023-04-23 05:53:59,482 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/aabcf50ab17a4896ad360bd4f358dbf2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/aabcf50ab17a4896ad360bd4f358dbf2 2023-04-23 05:53:59,489 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/aabcf50ab17a4896ad360bd4f358dbf2, entries=150, sequenceid=199, filesize=11.8 K 2023-04-23 05:53:59,490 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bfd1e7528fb104543b2ee708b991f02a in 949ms, sequenceid=199, compaction requested=true 2023-04-23 05:53:59,490 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:53:59,490 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:59,491 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:53:59,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:53:59,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:53:59,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:53:59,494 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:59,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:59,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:53:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:59,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:53:59,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d02c153[Count = 0] remaining members to acquire global barrier 2023-04-23 05:53:59,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:53:59,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:59,499 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:59,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:59,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:53:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:53:59,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:53:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:53:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:53:59,503 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:53:59,506 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:53:59,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:53:59,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:53:59,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:53:59,506 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:53:59,509 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:53:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:53:59,509 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:59,509 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:59,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:59,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:59,510 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:53:59,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:53:59,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:59,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:53:59,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:53:59,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:59,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:53:59,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:53:59,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,515 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:59,515 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:59,515 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,515 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:59,515 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:53:59,515 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:59,515 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:53:59,515 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:53:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:53:59,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:53:59,615 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:53:59,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:53:59,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:53:59,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:53:59,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:53:59,628 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:53:59,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:59,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:53:59,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:53:59,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:59,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:59,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:53:59,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:53:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:53:59,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:53:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:59,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:53:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:53:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:53:59,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:53:59,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:53:59,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:53:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:53:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:53:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:53:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:53:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:53:59,636 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:53:59,636 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:53:59,637 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:53:59,637 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:53:59,637 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:59,637 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:53:59,637 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:59,637 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:53:59,637 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:53:59,663 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/94b543d6ed98418bb2d0dddf1f9169e5 2023-04-23 05:53:59,700 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/a47edd96442144d1a62a46f4d3689c53 2023-04-23 05:53:59,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:53:59,755 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:53:59,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:59,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229299768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:59,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229299768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:59,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229299770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:59,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229299874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:59,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229299875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:53:59,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:53:59,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229299875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229300081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229300081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229300081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,119 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/44523bd8e71e446a87bbf94d5f71a3ac 2023-04-23 05:54:00,129 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/94b543d6ed98418bb2d0dddf1f9169e5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/94b543d6ed98418bb2d0dddf1f9169e5 2023-04-23 05:54:00,140 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/94b543d6ed98418bb2d0dddf1f9169e5, entries=150, sequenceid=219, filesize=11.8 K 2023-04-23 05:54:00,142 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/a47edd96442144d1a62a46f4d3689c53 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a47edd96442144d1a62a46f4d3689c53 2023-04-23 05:54:00,160 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a47edd96442144d1a62a46f4d3689c53, entries=150, sequenceid=219, filesize=11.8 K 2023-04-23 05:54:00,163 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/44523bd8e71e446a87bbf94d5f71a3ac as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/44523bd8e71e446a87bbf94d5f71a3ac 2023-04-23 05:54:00,173 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/44523bd8e71e446a87bbf94d5f71a3ac, entries=150, sequenceid=219, filesize=11.8 K 2023-04-23 05:54:00,174 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bfd1e7528fb104543b2ee708b991f02a in 538ms, sequenceid=219, compaction requested=true 2023-04-23 05:54:00,174 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:00,174 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:00,174 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:00,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:00,178 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,178 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,178 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,178 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:00,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@621b32b0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:00,182 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:00,182 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,185 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:00,186 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:00,186 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:00,186 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,186 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:00,187 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:00,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:00,191 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:00,191 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:00,191 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:00,191 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:00,191 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:00,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,194 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,197 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:00,207 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:00,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,207 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:00,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:00,207 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,208 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:00,308 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:00,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:00,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:00,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:00,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:00,332 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:00,332 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:00,332 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:00,332 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:00,332 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,334 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:00,334 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:00,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:00,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:00,349 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:00,349 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:00,349 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:00,356 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:00,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:00,368 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:00,368 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:00,368 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:54:00,369 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:00,369 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:00,369 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:00,369 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:00,369 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:00,369 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:00,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:00,393 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:00,410 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/03e447eaad194a79ad27f6b22f03b1a3 2023-04-23 05:54:00,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229240424, max timestamp=1682229240442Num uniques 859; 2023-04-23 05:54:00,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:00,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229240424, max timestamp=1682229240442Num uniques 859; 2023-04-23 05:54:00,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:00,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229240424, max timestamp=1682229240442Num uniques 859; 2023-04-23 05:54:00,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:00,481 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/832c8fff0d674172851b1e486ea1c6a4 2023-04-23 05:54:00,511 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/722b721da1f34b5c94461a8e45c1ca27 2023-04-23 05:54:00,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229300515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,520 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/03e447eaad194a79ad27f6b22f03b1a3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/03e447eaad194a79ad27f6b22f03b1a3 2023-04-23 05:54:00,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56582 deadline: 1682229300518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229300519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,528 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/03e447eaad194a79ad27f6b22f03b1a3, entries=150, sequenceid=231, filesize=11.8 K 2023-04-23 05:54:00,530 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/832c8fff0d674172851b1e486ea1c6a4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/832c8fff0d674172851b1e486ea1c6a4 2023-04-23 05:54:00,539 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/832c8fff0d674172851b1e486ea1c6a4, entries=150, sequenceid=231, filesize=11.8 K 2023-04-23 05:54:00,540 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/722b721da1f34b5c94461a8e45c1ca27 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/722b721da1f34b5c94461a8e45c1ca27 2023-04-23 05:54:00,546 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/722b721da1f34b5c94461a8e45c1ca27, entries=150, sequenceid=231, filesize=11.8 K 2023-04-23 05:54:00,547 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for bfd1e7528fb104543b2ee708b991f02a in 179ms, sequenceid=231, compaction requested=true 2023-04-23 05:54:00,547 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:00,547 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:00,547 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:00,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:00,554 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,554 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:00,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:00,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34b91fbb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:00,571 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:00,571 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:00,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:00,576 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:00,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:00,579 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:00,579 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:00,579 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:00,579 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:00,579 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:00,581 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,581 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,581 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:00,582 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,585 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,587 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:00,587 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:00,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:00,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,588 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,588 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,588 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:00,588 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:00,588 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,588 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:00,588 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,589 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,589 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:00,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:00,625 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-23 05:54:00,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:00,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:00,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:00,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:00,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:00,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:00,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229300642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229300645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229300647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,653 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/5a8f199c37e344d7abdfe7b3b19a7394 2023-04-23 05:54:00,688 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:00,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:00,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:00,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:00,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:00,693 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:00,693 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:00,693 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:00,693 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:00,694 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,694 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:00,695 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:00,696 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,696 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:00,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:00,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:00,698 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:00,698 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:00,698 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:00,698 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:00,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,704 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,705 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,705 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:00,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6985c4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:00,706 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:00,706 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,707 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,708 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:00,708 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:00,708 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:00,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:00,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,709 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,709 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,710 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:00,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:00,712 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:00,712 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:00,712 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:00,712 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:00,712 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:00,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,715 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,716 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,719 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:00,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:00,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:00,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,720 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,720 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,720 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,720 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:00,720 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:00,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:00,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:00,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229300751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229300751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229300753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,820 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:00,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:00,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:00,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:00,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:00,830 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:00,830 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:00,830 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:00,830 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:00,830 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,831 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,832 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:00,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:00,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:00,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:00,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:00,834 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:00,834 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:00,834 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:00,834 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:00,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:00,835 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,835 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,836 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,836 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:00,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aacd83[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:00,838 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:00,838 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,839 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:00,840 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:00,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:00,843 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:00,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:00,846 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:00,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:00,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:00,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:00,846 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:00,848 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,848 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,848 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,849 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,851 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:00,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:00,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:00,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:00,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,858 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,858 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:00,858 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:00,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:00,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:00,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56650 deadline: 1682229300954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:00,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:00,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229300955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:00,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:00,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229300957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:00,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:00,964 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:00,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:00,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:00,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:00,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:00,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:00,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:00,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:00,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:00,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:00,968 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:00,968 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:00,968 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:00,968 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:00,968 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:00,973 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,973 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,974 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,974 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:00,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a9ff1b4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:00,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:00,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,976 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,976 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:00,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:00,977 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:00,977 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:00,978 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:00,981 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:00,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:00,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:00,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:00,981 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:00,983 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,983 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:00,983 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,984 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:00,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:00,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:00,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:00,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:00,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:00,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:00,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:00,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:00,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:00,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:00,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:00,990 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:00,990 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:00,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:01,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/b383fd28dc2741349ab7a3fdf82b63b2 2023-04-23 05:54:01,090 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:01,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:01,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:01,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/62280dccd07e4d608a6dd2de70f94404 2023-04-23 05:54:01,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:01,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:01,095 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:01,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:01,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:01,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:01,096 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,096 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,097 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:01,097 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:01,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,099 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,099 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:01,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:01,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:01,100 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:01,100 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:01,100 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:01,100 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:01,100 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:01,100 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:01,101 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:01,101 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:01,101 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:01,101 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:01,101 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:01,101 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,101 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:01,101 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:01,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/5a8f199c37e344d7abdfe7b3b19a7394 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5a8f199c37e344d7abdfe7b3b19a7394 2023-04-23 05:54:01,102 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:01,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:01,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:01,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:01,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:01,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46a7659b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:01,105 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:01,105 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,107 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,107 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:01,107 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:01,107 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:01,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:01,109 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:01,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:01,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5a8f199c37e344d7abdfe7b3b19a7394, entries=200, sequenceid=267, filesize=14.3 K 2023-04-23 05:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:01,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/b383fd28dc2741349ab7a3fdf82b63b2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b383fd28dc2741349ab7a3fdf82b63b2 2023-04-23 05:54:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:01,114 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:01,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:01,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:01,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:01,114 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:01,116 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,116 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:01,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,117 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:01,118 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b383fd28dc2741349ab7a3fdf82b63b2, entries=150, sequenceid=267, filesize=11.9 K 2023-04-23 05:54:01,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/62280dccd07e4d608a6dd2de70f94404 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/62280dccd07e4d608a6dd2de70f94404 2023-04-23 05:54:01,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/62280dccd07e4d608a6dd2de70f94404, entries=150, sequenceid=267, filesize=11.9 K 2023-04-23 05:54:01,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for bfd1e7528fb104543b2ee708b991f02a in 510ms, sequenceid=267, compaction requested=true 2023-04-23 05:54:01,136 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:01,136 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:01,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:01,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:01,136 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:01,136 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:01,136 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:01,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:01,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:01,136 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,136 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:01,136 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,136 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:01,136 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:54:01,136 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:54:01,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:01,137 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:01,136 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,137 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,137 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,137 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,137 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,139 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72708 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:54:01,139 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/B is initiating minor compaction (all files) 2023-04-23 05:54:01,139 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/B in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:01,140 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/aabd230b48464c2fa6a7b5b3554cbf4e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/35f032e7857a4548aefadde93266984f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/dc104c27820c45f783ea087c5b52887d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a47edd96442144d1a62a46f4d3689c53, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/832c8fff0d674172851b1e486ea1c6a4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b383fd28dc2741349ab7a3fdf82b63b2] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=71.0 K 2023-04-23 05:54:01,140 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 77538 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:54:01,140 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/A is initiating minor compaction (all files) 2023-04-23 05:54:01,140 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/A in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:01,140 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/69a21fbb7bad401ab1b9d14a32a98313, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/e7d210f892df4ee0815909adf9dca9ce, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5f7938b5ac3f47c99df255d9f07269f0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/94b543d6ed98418bb2d0dddf1f9169e5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/03e447eaad194a79ad27f6b22f03b1a3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5a8f199c37e344d7abdfe7b3b19a7394] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=75.7 K 2023-04-23 05:54:01,141 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting aabd230b48464c2fa6a7b5b3554cbf4e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682229236157 2023-04-23 05:54:01,141 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 69a21fbb7bad401ab1b9d14a32a98313, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682229236157 2023-04-23 05:54:01,150 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e7d210f892df4ee0815909adf9dca9ce, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1682229236808 2023-04-23 05:54:01,150 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 35f032e7857a4548aefadde93266984f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1682229236808 2023-04-23 05:54:01,151 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 5f7938b5ac3f47c99df255d9f07269f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682229237943 2023-04-23 05:54:01,151 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting dc104c27820c45f783ea087c5b52887d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682229237943 2023-04-23 05:54:01,151 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 94b543d6ed98418bb2d0dddf1f9169e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1682229238627 2023-04-23 05:54:01,152 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 03e447eaad194a79ad27f6b22f03b1a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682229239764 2023-04-23 05:54:01,152 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a47edd96442144d1a62a46f4d3689c53, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1682229238627 2023-04-23 05:54:01,153 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 5a8f199c37e344d7abdfe7b3b19a7394, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682229240512 2023-04-23 05:54:01,153 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 832c8fff0d674172851b1e486ea1c6a4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682229239764 2023-04-23 05:54:01,153 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b383fd28dc2741349ab7a3fdf82b63b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682229240512 2023-04-23 05:54:01,180 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#A#compaction#48 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:01,190 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#B#compaction#49 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:01,210 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/d77d5a12f6ed41c6a81142735247579b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d77d5a12f6ed41c6a81142735247579b 2023-04-23 05:54:01,222 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/3c71756928384aa08621dfc35354675d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/3c71756928384aa08621dfc35354675d 2023-04-23 05:54:01,225 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/A of bfd1e7528fb104543b2ee708b991f02a into d77d5a12f6ed41c6a81142735247579b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:01,225 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:01,225 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/A, priority=10, startTime=1682229241136; duration=0sec 2023-04-23 05:54:01,225 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:01,225 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:54:01,229 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72708 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:54:01,229 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/C is initiating minor compaction (all files) 2023-04-23 05:54:01,229 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/C in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:01,229 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/b412b4919e9f4f13b0d1aa27dd0a0f36, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a1919cdbc76e4028ab7bd6538902627f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/aabcf50ab17a4896ad360bd4f358dbf2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/44523bd8e71e446a87bbf94d5f71a3ac, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/722b721da1f34b5c94461a8e45c1ca27, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/62280dccd07e4d608a6dd2de70f94404] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=71.0 K 2023-04-23 05:54:01,229 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b412b4919e9f4f13b0d1aa27dd0a0f36, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682229236157 2023-04-23 05:54:01,231 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/B of bfd1e7528fb104543b2ee708b991f02a into 3c71756928384aa08621dfc35354675d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:01,232 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:01,232 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/B, priority=10, startTime=1682229241136; duration=0sec 2023-04-23 05:54:01,232 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:01,237 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:01,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:01,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:01,241 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a1919cdbc76e4028ab7bd6538902627f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1682229236808 2023-04-23 05:54:01,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:01,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:01,242 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:01,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:01,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:01,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:01,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,243 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting aabcf50ab17a4896ad360bd4f358dbf2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682229237943 2023-04-23 05:54:01,245 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 44523bd8e71e446a87bbf94d5f71a3ac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1682229238627 2023-04-23 05:54:01,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:01,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:01,245 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 722b721da1f34b5c94461a8e45c1ca27, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682229239764 2023-04-23 05:54:01,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,246 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 62280dccd07e4d608a6dd2de70f94404, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682229240512 2023-04-23 05:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:01,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:01,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:01,260 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:01,260 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:01,260 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:01,260 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:01,260 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:01,261 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:01,261 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:01,261 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:54:01,262 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:01,263 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:01,263 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:01,265 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:01,265 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:01,265 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:01,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:01,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:01,270 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#C#compaction#50 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:01,320 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/cf1495f044f4404685b749b5f71b4bf1 2023-04-23 05:54:01,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229241290, max timestamp=1682229241306Num uniques 859; 2023-04-23 05:54:01,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229241290, max timestamp=1682229241306Num uniques 859; 2023-04-23 05:54:01,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:54:01,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:54:01,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682229241290, max timestamp=1682229241306Num uniques 859; 2023-04-23 05:54:01,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-23 05:54:01,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:01,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229301386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,398 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/83046b576e4844198d84fb2e4f1c94a6 2023-04-23 05:54:01,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:01,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229301390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:01,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229301391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229301496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:01,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229301500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:01,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229301505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:01,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229301698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,707 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/96b16cf73355446dbc7ed40170d238d1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/96b16cf73355446dbc7ed40170d238d1 2023-04-23 05:54:01,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:01,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229301707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:01,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229301709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,714 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/C of bfd1e7528fb104543b2ee708b991f02a into 96b16cf73355446dbc7ed40170d238d1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:01,714 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:01,714 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/C, priority=10, startTime=1682229241136; duration=0sec 2023-04-23 05:54:01,714 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:01,842 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/879451fd07d24cff8ad94657ebb6ef43 2023-04-23 05:54:01,849 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/cf1495f044f4404685b749b5f71b4bf1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cf1495f044f4404685b749b5f71b4bf1 2023-04-23 05:54:01,857 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cf1495f044f4404685b749b5f71b4bf1, entries=150, sequenceid=280, filesize=11.9 K 2023-04-23 05:54:01,859 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/83046b576e4844198d84fb2e4f1c94a6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/83046b576e4844198d84fb2e4f1c94a6 2023-04-23 05:54:01,863 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/83046b576e4844198d84fb2e4f1c94a6, entries=150, sequenceid=280, filesize=11.9 K 2023-04-23 05:54:01,864 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/879451fd07d24cff8ad94657ebb6ef43 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/879451fd07d24cff8ad94657ebb6ef43 2023-04-23 05:54:01,870 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/879451fd07d24cff8ad94657ebb6ef43, entries=150, sequenceid=280, filesize=11.9 K 2023-04-23 05:54:01,871 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=124.84 KB/127835 for bfd1e7528fb104543b2ee708b991f02a in 610ms, sequenceid=280, compaction requested=false 2023-04-23 05:54:01,871 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:01,871 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:01,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:01,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:01,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:01,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:01,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:01,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40669f2e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:01,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:01,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,879 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:01,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:01,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:01,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:01,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:01,883 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:01,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:01,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:01,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:01,884 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:01,885 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,885 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,885 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:01,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:01,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:01,886 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,888 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:01,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:01,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:01,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:01,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:01,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:01,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:01,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:01,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:01,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:01,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:01,892 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:01,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:01,893 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:01,993 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:01,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:01,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:02,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:02,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:02,000 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:02,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:02,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:02,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,004 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,004 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:02,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:02,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:02,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:02,013 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-04-23 05:54:02,013 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:02,013 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:02,013 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:02,013 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:02,013 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:02,013 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:02,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:02,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:02,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:02,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:02,027 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,027 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:02,027 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:02,027 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:02,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,032 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,032 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:02,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f78474[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:02,036 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:02,036 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229302034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229302036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,040 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229302036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,041 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,041 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:02,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:02,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:02,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:02,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:02,043 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,042 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,043 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:02,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:02,049 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:02,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:02,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:02,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:02,049 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:02,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,052 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,063 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,068 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,068 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,068 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,069 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:02,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:02,069 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/deae3002a4644ac9914c543f091e0bac 2023-04-23 05:54:02,070 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:02,070 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:02,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:02,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/4d6b6ac15b6d43d38683d84365c7e43f 2023-04-23 05:54:02,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229302142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229302142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229302143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,170 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:02,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:02,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:02,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:02,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:02,200 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:02,200 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,200 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:02,200 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:02,201 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,201 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,204 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,204 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,204 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,204 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:02,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,218 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:02,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:02,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:02,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:02,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:02,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:02,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:02,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:02,235 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:02,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52a0c56f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:02,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:02,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,239 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:02,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:02,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:02,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:02,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:02,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:02,246 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:02,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:02,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:02,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:02,247 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:02,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,251 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,253 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,256 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,256 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,256 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,256 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:02,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:02,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:02,256 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,257 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:02,257 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:02,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229302347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229302347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229302348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,357 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:02,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:02,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:02,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:02,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:02,366 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:02,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:02,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:02,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,368 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,369 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,369 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:02,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:02,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:02,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:02,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:02,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:02,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:02,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:02,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:02,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:02,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:02,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:02,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,381 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,381 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,381 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:02,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fd6f91d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:02,384 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:02,384 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,387 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:02,387 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:02,387 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,389 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:02,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:02,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:02,389 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:02,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:02,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:02,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:02,389 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:02,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,400 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,402 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,405 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,405 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,405 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,405 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:02,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:02,406 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,406 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:02,406 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:02,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:02,506 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:02,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:02,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:02,510 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:02,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:02,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:02,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,513 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,516 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,516 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:02,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:02,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:02,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:02,517 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:02,517 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,517 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:02,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:02,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:02,518 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,518 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:02,518 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:02,518 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:02,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:02,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:02,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:02,520 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:02,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:02,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b3d9c26[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:02,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:02,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,524 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:02,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,525 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:02,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:02,530 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,530 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,530 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,530 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:02,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:02,533 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:02,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:02,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:02,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:02,533 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:02,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,536 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,539 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,543 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,543 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,543 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,543 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,543 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,543 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,543 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:02,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:02,544 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,543 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,544 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,544 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:02,544 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:02,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/e5e675b55b974d898ef74ee1f71070d4 2023-04-23 05:54:02,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/deae3002a4644ac9914c543f091e0bac as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/deae3002a4644ac9914c543f091e0bac 2023-04-23 05:54:02,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/deae3002a4644ac9914c543f091e0bac, entries=200, sequenceid=317, filesize=14.3 K 2023-04-23 05:54:02,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/4d6b6ac15b6d43d38683d84365c7e43f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/4d6b6ac15b6d43d38683d84365c7e43f 2023-04-23 05:54:02,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/4d6b6ac15b6d43d38683d84365c7e43f, entries=150, sequenceid=317, filesize=11.9 K 2023-04-23 05:54:02,639 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/e5e675b55b974d898ef74ee1f71070d4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e5e675b55b974d898ef74ee1f71070d4 2023-04-23 05:54:02,644 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:02,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:02,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:02,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229302653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229302654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:02,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229302654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:02,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:02,660 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:02,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:02,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:02,661 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,661 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,662 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,662 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,663 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,663 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:02,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:02,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:02,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:02,664 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:02,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:02,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:02,668 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,668 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:02,669 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:02,669 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:02,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:02,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:02,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:02,672 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,673 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,673 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,673 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:02,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e5e675b55b974d898ef74ee1f71070d4, entries=150, sequenceid=317, filesize=11.9 K 2023-04-23 05:54:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134705, heapSize ~374.26 KB/383240, currentSize=46.96 KB/48090 for bfd1e7528fb104543b2ee708b991f02a in 663ms, sequenceid=317, compaction requested=true 2023-04-23 05:54:02,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:02,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:02,675 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:02,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:02,675 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:02,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:02,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70427711[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:02,676 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:02,676 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,677 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:02,677 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/A is initiating minor compaction (all files) 2023-04-23 05:54:02,677 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/A in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,677 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d77d5a12f6ed41c6a81142735247579b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cf1495f044f4404685b749b5f71b4bf1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/deae3002a4644ac9914c543f091e0bac] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=38.6 K 2023-04-23 05:54:02,678 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,678 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:02,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,678 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/B is initiating minor compaction (all files) 2023-04-23 05:54:02,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:02,678 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/B in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:02,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:02,678 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting d77d5a12f6ed41c6a81142735247579b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682229240512 2023-04-23 05:54:02,678 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/3c71756928384aa08621dfc35354675d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/83046b576e4844198d84fb2e4f1c94a6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/4d6b6ac15b6d43d38683d84365c7e43f] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=36.2 K 2023-04-23 05:54:02,679 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting cf1495f044f4404685b749b5f71b4bf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682229240626 2023-04-23 05:54:02,679 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3c71756928384aa08621dfc35354675d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682229240512 2023-04-23 05:54:02,680 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting deae3002a4644ac9914c543f091e0bac, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682229241376 2023-04-23 05:54:02,680 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 83046b576e4844198d84fb2e4f1c94a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682229240626 2023-04-23 05:54:02,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:02,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,681 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 4d6b6ac15b6d43d38683d84365c7e43f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682229241376 2023-04-23 05:54:02,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:02,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:02,684 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:02,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:02,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:02,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:02,684 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:02,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:02,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,690 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:02,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:02,694 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#A#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,694 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,694 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:02,694 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:02,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:02,702 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#B#compaction#58 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:02,734 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/268145bea74f48148f92b4f076c18b91 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/268145bea74f48148f92b4f076c18b91 2023-04-23 05:54:02,735 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/c63c227c182f4666a0eaf2bb4f4a250e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c63c227c182f4666a0eaf2bb4f4a250e 2023-04-23 05:54:02,748 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/A of bfd1e7528fb104543b2ee708b991f02a into 268145bea74f48148f92b4f076c18b91(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:02,748 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:02,748 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/A, priority=13, startTime=1682229242675; duration=0sec 2023-04-23 05:54:02,748 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:02,748 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:02,750 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/B of bfd1e7528fb104543b2ee708b991f02a into c63c227c182f4666a0eaf2bb4f4a250e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:02,750 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:02,750 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/B, priority=13, startTime=1682229242675; duration=0sec 2023-04-23 05:54:02,750 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:02,752 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:02,752 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/C is initiating minor compaction (all files) 2023-04-23 05:54:02,752 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/C in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,752 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/96b16cf73355446dbc7ed40170d238d1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/879451fd07d24cff8ad94657ebb6ef43, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e5e675b55b974d898ef74ee1f71070d4] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=36.2 K 2023-04-23 05:54:02,753 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 96b16cf73355446dbc7ed40170d238d1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682229240512 2023-04-23 05:54:02,753 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 879451fd07d24cff8ad94657ebb6ef43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1682229240626 2023-04-23 05:54:02,754 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e5e675b55b974d898ef74ee1f71070d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682229241376 2023-04-23 05:54:02,777 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#C#compaction#59 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:02,795 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:02,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:02,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:02,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:02,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:02,799 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:02,799 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:02,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:02,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,802 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:02,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:02,802 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:02,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:02,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:02,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:02,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:02,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:02,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:02,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:02,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:02,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:02,812 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:02,812 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:02,813 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:54:02,813 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:02,813 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:02,813 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:02,813 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:02,813 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:02,813 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:02,841 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/4214df568174487188b4952d49fbced2 2023-04-23 05:54:03,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:03,159 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682229243170, max timestamp=1682229243181Num uniques 859; 2023-04-23 05:54:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-23 05:54:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229243170, max timestamp=1682229243182Num uniques 859; 2023-04-23 05:54:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682229243170, max timestamp=1682229243181Num uniques 859; 2023-04-23 05:54:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-23 05:54:03,196 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/e686ff83ce264ed5b51d5f2619b21143 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e686ff83ce264ed5b51d5f2619b21143 2023-04-23 05:54:03,217 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/C of bfd1e7528fb104543b2ee708b991f02a into e686ff83ce264ed5b51d5f2619b21143(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:03,217 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:03,217 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/C, priority=13, startTime=1682229242675; duration=0sec 2023-04-23 05:54:03,217 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:03,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229303258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682229243199, max timestamp=1682229243256Num uniques 1009; 2023-04-23 05:54:03,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-23 05:54:03,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682229243199, max timestamp=1682229243256Num uniques 1018; 2023-04-23 05:54:03,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-23 05:54:03,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682229243199, max timestamp=1682229243256Num uniques 1009; 2023-04-23 05:54:03,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-04-23 05:54:03,286 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/ecac09eeb0b34ec3a08c4893a4567b2c 2023-04-23 05:54:03,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229303306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229303306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,314 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/dec89b249c624cf7be25969fb715798d 2023-04-23 05:54:03,320 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/4214df568174487188b4952d49fbced2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4214df568174487188b4952d49fbced2 2023-04-23 05:54:03,329 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4214df568174487188b4952d49fbced2, entries=100, sequenceid=329, filesize=9.5 K 2023-04-23 05:54:03,330 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/ecac09eeb0b34ec3a08c4893a4567b2c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/ecac09eeb0b34ec3a08c4893a4567b2c 2023-04-23 05:54:03,340 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/ecac09eeb0b34ec3a08c4893a4567b2c, entries=100, sequenceid=329, filesize=9.5 K 2023-04-23 05:54:03,341 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/dec89b249c624cf7be25969fb715798d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/dec89b249c624cf7be25969fb715798d 2023-04-23 05:54:03,353 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/dec89b249c624cf7be25969fb715798d, entries=100, sequenceid=329, filesize=9.5 K 2023-04-23 05:54:03,354 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bfd1e7528fb104543b2ee708b991f02a in 542ms, sequenceid=329, compaction requested=false 2023-04-23 05:54:03,355 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:03,355 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:03,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:03,362 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,362 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:03,362 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,362 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:03,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:03,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@249d4fcb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:03,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:03,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,367 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:03,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:03,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:03,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:03,367 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-04-23 05:54:03,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:03,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:03,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:03,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:03,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:03,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:03,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:03,372 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:03,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:03,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:03,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:03,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:03,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:03,376 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:03,376 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:03,376 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:03,376 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:03,376 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:03,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:03,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,381 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,383 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,384 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,386 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:03,386 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:03,386 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:03,386 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,386 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,386 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,386 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,386 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,386 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:03,386 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:03,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:03,387 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:03,387 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:03,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:03,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:03,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a030b4dac16c4f158ad00308acbd0d1d 2023-04-23 05:54:03,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229303398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229303409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229303410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,487 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:03,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:03,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:03,496 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,498 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:03,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,499 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,499 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:03,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:03,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:03,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:03,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:03,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:03,501 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,501 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:03,501 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:03,501 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:03,504 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,505 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:03,508 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:03,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:03,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229303508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c9b4454[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:03,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:03,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,512 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:03,513 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,513 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:03,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:03,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:03,520 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:03,520 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:03,520 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:03,520 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:03,520 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:03,525 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,525 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:03,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,525 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:03,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,526 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:03,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:03,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:03,535 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:03,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:03,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:03,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:03,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,535 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:03,535 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:03,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:03,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229303615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229303615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,636 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:03,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:03,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:03,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:03,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:03,639 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:03,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:03,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:03,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:03,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:03,641 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,642 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,642 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:03,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:03,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:03,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:03,644 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,644 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:03,644 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:03,644 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:03,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:03,646 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,646 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:03,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,647 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:03,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:03,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b70334d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:03,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:03,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,649 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:03,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:03,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:03,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:03,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:03,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:03,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:03,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:03,655 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:03,655 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:03,655 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:03,655 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:03,655 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:03,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:03,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229303711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,723 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,723 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,723 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:03,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:03,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,735 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,815 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:03,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:03,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:03,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:03,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:03,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:03,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:03,816 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:03,816 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:03,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:03,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/9e276a45e91b407bafb26da64ee84c8f 2023-04-23 05:54:03,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/3d2630aec2094f24ac805fcc0a9299c2 2023-04-23 05:54:03,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a030b4dac16c4f158ad00308acbd0d1d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a030b4dac16c4f158ad00308acbd0d1d 2023-04-23 05:54:03,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a030b4dac16c4f158ad00308acbd0d1d, entries=200, sequenceid=384, filesize=14.3 K 2023-04-23 05:54:03,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/9e276a45e91b407bafb26da64ee84c8f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9e276a45e91b407bafb26da64ee84c8f 2023-04-23 05:54:03,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9e276a45e91b407bafb26da64ee84c8f, entries=150, sequenceid=384, filesize=11.9 K 2023-04-23 05:54:03,859 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/3d2630aec2094f24ac805fcc0a9299c2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/3d2630aec2094f24ac805fcc0a9299c2 2023-04-23 05:54:03,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/3d2630aec2094f24ac805fcc0a9299c2, entries=150, sequenceid=384, filesize=11.9 K 2023-04-23 05:54:03,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=40.25 KB/41220 for bfd1e7528fb104543b2ee708b991f02a in 508ms, sequenceid=384, compaction requested=true 2023-04-23 05:54:03,876 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:03,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:03,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-23 05:54:03,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-23 05:54:03,876 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:03,877 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:03,878 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37202 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:03,878 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/A is initiating minor compaction (all files) 2023-04-23 05:54:03,878 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/A in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,878 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:03,878 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/268145bea74f48148f92b4f076c18b91, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4214df568174487188b4952d49fbced2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a030b4dac16c4f158ad00308acbd0d1d] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=36.3 K 2023-04-23 05:54:03,878 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/C is initiating minor compaction (all files) 2023-04-23 05:54:03,878 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/C in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,878 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e686ff83ce264ed5b51d5f2619b21143, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/dec89b249c624cf7be25969fb715798d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/3d2630aec2094f24ac805fcc0a9299c2] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=33.9 K 2023-04-23 05:54:03,879 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 268145bea74f48148f92b4f076c18b91, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682229241376 2023-04-23 05:54:03,880 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e686ff83ce264ed5b51d5f2619b21143, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682229241376 2023-04-23 05:54:03,880 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 4214df568174487188b4952d49fbced2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682229242017 2023-04-23 05:54:03,881 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting dec89b249c624cf7be25969fb715798d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682229242017 2023-04-23 05:54:03,881 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a030b4dac16c4f158ad00308acbd0d1d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682229243290 2023-04-23 05:54:03,882 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3d2630aec2094f24ac805fcc0a9299c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682229243290 2023-04-23 05:54:03,893 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#C#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:03,902 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#A#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:03,916 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:03,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:03,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:03,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:03,927 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:54:03,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:03,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:03,929 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/4a8d33552f134d0b9e8ad3e7a7e6a8c1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4a8d33552f134d0b9e8ad3e7a7e6a8c1 2023-04-23 05:54:03,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:03,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:03,929 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:03,929 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:03,938 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/A of bfd1e7528fb104543b2ee708b991f02a into 4a8d33552f134d0b9e8ad3e7a7e6a8c1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:03,939 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:03,939 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/A, priority=13, startTime=1682229243876; duration=0sec 2023-04-23 05:54:03,939 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:03,939 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:03,940 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:03,940 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:03,940 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:03,940 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:03,940 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:03,941 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/B is initiating minor compaction (all files) 2023-04-23 05:54:03,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,941 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/B in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,942 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c63c227c182f4666a0eaf2bb4f4a250e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/ecac09eeb0b34ec3a08c4893a4567b2c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9e276a45e91b407bafb26da64ee84c8f] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=33.9 K 2023-04-23 05:54:03,943 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c63c227c182f4666a0eaf2bb4f4a250e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682229241376 2023-04-23 05:54:03,943 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting ecac09eeb0b34ec3a08c4893a4567b2c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682229242017 2023-04-23 05:54:03,944 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 9e276a45e91b407bafb26da64ee84c8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682229243290 2023-04-23 05:54:03,952 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/5ed8e65a805747319ce667f7c57fd9bb as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5ed8e65a805747319ce667f7c57fd9bb 2023-04-23 05:54:03,963 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/C of bfd1e7528fb104543b2ee708b991f02a into 5ed8e65a805747319ce667f7c57fd9bb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:03,963 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:03,963 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/C, priority=13, startTime=1682229243876; duration=0sec 2023-04-23 05:54:03,963 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:03,969 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#B#compaction#69 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:03,984 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:03,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:03,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:03,985 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/e49ab6c1c6c645e69b99c192aa37faaf as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/e49ab6c1c6c645e69b99c192aa37faaf 2023-04-23 05:54:03,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:03,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:03,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:03,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229243958, max timestamp=1682229243980Num uniques 859; 2023-04-23 05:54:03,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:03,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229243958, max timestamp=1682229243979Num uniques 859; 2023-04-23 05:54:03,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:03,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:03,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229243958, max timestamp=1682229243979Num uniques 859; 2023-04-23 05:54:03,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:03,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:03,988 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,988 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:03,988 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:03,988 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:03,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:03,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:03,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:03,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:03,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:03,992 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/B of bfd1e7528fb104543b2ee708b991f02a into e49ab6c1c6c645e69b99c192aa37faaf(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:03,992 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:03,993 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/B, priority=13, startTime=1682229243876; duration=0sec 2023-04-23 05:54:03,993 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:04,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229304013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229304014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229304017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:04,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@534229cb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:04,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:04,055 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:04,055 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,055 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,055 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:04,115 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:04,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:04,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:04,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:04,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229304119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229304120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,169 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:04,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:04,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:04,172 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:04,172 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:04,172 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:04,172 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:04,172 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:04,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:04,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:04,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,299 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:04,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:04,300 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:04,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:04,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:04,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:04,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:04,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:04,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:04,300 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:04,300 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:04,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:04,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:04,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:04,301 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,301 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,301 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229304323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229304324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,400 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:04,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:04,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:04,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:04,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:04,404 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:04,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:04,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:04,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:04,405 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,405 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:04,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/d7946e18f3ab4d3eb5f2e93850115c3b 2023-04-23 05:54:04,438 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:04,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:04,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:04,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:04,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:04,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:04,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:04,440 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:04,441 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:04,441 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:04,441 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:04,441 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:04,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:04,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:04,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:04,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:04,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e937cd[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:04,509 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:04,509 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229304523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,584 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,584 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:04,584 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:04,584 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,585 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:04,585 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:04,615 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:04,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:04,629 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:04,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:04,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:04,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:04,630 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:04,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229304630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:04,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229304630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,684 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,684 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:04,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,684 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:04,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:04,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,687 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:04,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/bd56f2b3042f4d45960ac35b45e502be 2023-04-23 05:54:04,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/87e69ff16391460c9e85f63de48b21f6 2023-04-23 05:54:04,872 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/d7946e18f3ab4d3eb5f2e93850115c3b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d7946e18f3ab4d3eb5f2e93850115c3b 2023-04-23 05:54:04,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d7946e18f3ab4d3eb5f2e93850115c3b, entries=200, sequenceid=395, filesize=14.3 K 2023-04-23 05:54:04,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/bd56f2b3042f4d45960ac35b45e502be as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/bd56f2b3042f4d45960ac35b45e502be 2023-04-23 05:54:04,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/bd56f2b3042f4d45960ac35b45e502be, entries=150, sequenceid=395, filesize=11.9 K 2023-04-23 05:54:04,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/87e69ff16391460c9e85f63de48b21f6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/87e69ff16391460c9e85f63de48b21f6 2023-04-23 05:54:04,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/87e69ff16391460c9e85f63de48b21f6, entries=150, sequenceid=395, filesize=11.9 K 2023-04-23 05:54:04,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for bfd1e7528fb104543b2ee708b991f02a in 969ms, sequenceid=395, compaction requested=false 2023-04-23 05:54:04,896 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:04,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:04,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:04,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:04,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,907 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:04,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:04,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:04,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:04,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:04,908 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:04,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:04,908 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:04,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:04,908 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:04,908 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:04,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:05,008 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:05,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:05,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:05,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:05,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:05,012 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:05,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:05,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:05,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:05,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:05,069 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:05,069 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,069 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:05,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:05,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:05,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:05,071 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:05,071 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:05,071 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:05,071 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:05,071 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:05,071 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:05,071 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:05,072 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-23 05:54:05,072 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:05,072 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:05,072 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:05,072 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:05,072 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:05,072 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:05,089 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/cab7221bafe04691b73717ec92638932 2023-04-23 05:54:05,122 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/09da67adfdf2489ca821de62846d5afc 2023-04-23 05:54:05,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:05,138 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:05,140 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/cc14f31466a843259f83bc32da29077c 2023-04-23 05:54:05,149 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/cab7221bafe04691b73717ec92638932 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cab7221bafe04691b73717ec92638932 2023-04-23 05:54:05,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229305153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229305154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,158 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cab7221bafe04691b73717ec92638932, entries=150, sequenceid=434, filesize=11.9 K 2023-04-23 05:54:05,159 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/09da67adfdf2489ca821de62846d5afc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/09da67adfdf2489ca821de62846d5afc 2023-04-23 05:54:05,165 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/09da67adfdf2489ca821de62846d5afc, entries=150, sequenceid=434, filesize=11.9 K 2023-04-23 05:54:05,167 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/cc14f31466a843259f83bc32da29077c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/cc14f31466a843259f83bc32da29077c 2023-04-23 05:54:05,173 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/cc14f31466a843259f83bc32da29077c, entries=150, sequenceid=434, filesize=11.9 K 2023-04-23 05:54:05,174 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for bfd1e7528fb104543b2ee708b991f02a in 102ms, sequenceid=434, compaction requested=true 2023-04-23 05:54:05,174 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:05,174 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:05,174 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:05,174 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:05,175 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,175 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:05,175 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:05,207 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:05,207 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,207 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:05,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:05,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26c4ce0c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:05,209 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:05,209 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:05,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:05,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:05,255 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,255 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:05,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:05,261 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-23 05:54:05,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:05,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:05,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:05,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:05,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:05,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:05,286 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/65f073c9dc55499f9802d015da5ad735 2023-04-23 05:54:05,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229305291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229305293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,307 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:05,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:05,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:05,311 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:05,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:05,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:05,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:05,311 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:05,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:05,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:05,362 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229305394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229305395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,401 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:05,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:05,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:05,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:05,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:05,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:05,402 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:05,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:05,402 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:05,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,402 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:05,502 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:05,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:05,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:05,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:05,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:05,505 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:05,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:05,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:05,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:05,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229305530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,546 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:05,546 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:05,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,546 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,546 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:05,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:05,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:05,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:05,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:05,547 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:05,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:05,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:05,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:05,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:05,548 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:05,548 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:05,548 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:05,548 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:05,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:05,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:05,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:05,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:05,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229305597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229305598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,608 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:05,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:05,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:05,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@409b8764[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:05,615 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:05,615 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:05,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:05,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:05,677 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,677 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:05,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/2b52745d66c64757acc9928db7bec9ea 2023-04-23 05:54:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:05,738 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:05,738 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:05,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:05,741 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:05,741 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:05,741 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:05,741 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:05,741 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:05,839 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,839 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,839 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:05,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:05,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,841 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:05,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:05,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:05,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229305901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229305902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:05,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:05,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:05,915 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:05,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:05,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:05,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:05,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,916 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:05,916 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:05,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:05,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:05,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:05,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:05,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:05,916 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:05,916 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:06,016 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:06,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:06,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:06,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:06,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:06,022 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:06,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:06,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:06,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:06,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,085 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:06,085 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:06,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:06,086 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,086 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:06,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:06,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:06,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:06,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:06,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:06,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:06,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:06,089 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:06,089 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:06,089 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:06,089 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:06,089 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:06,089 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:06,089 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:06,089 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:06,089 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,089 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:06,089 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:06,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/1af43ab28441444c9b044817c8d5d447 2023-04-23 05:54:06,134 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/65f073c9dc55499f9802d015da5ad735 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/65f073c9dc55499f9802d015da5ad735 2023-04-23 05:54:06,138 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,138 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:06,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:06,139 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:06,139 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:06,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:06,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:06,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:06,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4afb3e61[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:06,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/65f073c9dc55499f9802d015da5ad735, entries=200, sequenceid=448, filesize=14.3 K 2023-04-23 05:54:06,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:06,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/2b52745d66c64757acc9928db7bec9ea as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/2b52745d66c64757acc9928db7bec9ea 2023-04-23 05:54:06,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/2b52745d66c64757acc9928db7bec9ea, entries=150, sequenceid=448, filesize=11.9 K 2023-04-23 05:54:06,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/1af43ab28441444c9b044817c8d5d447 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1af43ab28441444c9b044817c8d5d447 2023-04-23 05:54:06,165 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1af43ab28441444c9b044817c8d5d447, entries=150, sequenceid=448, filesize=11.9 K 2023-04-23 05:54:06,166 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for bfd1e7528fb104543b2ee708b991f02a in 905ms, sequenceid=448, compaction requested=true 2023-04-23 05:54:06,166 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:06,167 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:06,167 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:54:06,167 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:54:06,167 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:06,167 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:06,169 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:54:06,169 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/B is initiating minor compaction (all files) 2023-04-23 05:54:06,169 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/B in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:06,169 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/e49ab6c1c6c645e69b99c192aa37faaf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/bd56f2b3042f4d45960ac35b45e502be, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/09da67adfdf2489ca821de62846d5afc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/2b52745d66c64757acc9928db7bec9ea] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=48.3 K 2023-04-23 05:54:06,169 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54374 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:54:06,170 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/A is initiating minor compaction (all files) 2023-04-23 05:54:06,170 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/A in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:06,170 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4a8d33552f134d0b9e8ad3e7a7e6a8c1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d7946e18f3ab4d3eb5f2e93850115c3b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cab7221bafe04691b73717ec92638932, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/65f073c9dc55499f9802d015da5ad735] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=53.1 K 2023-04-23 05:54:06,170 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e49ab6c1c6c645e69b99c192aa37faaf, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682229243290 2023-04-23 05:54:06,170 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 4a8d33552f134d0b9e8ad3e7a7e6a8c1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682229243290 2023-04-23 05:54:06,171 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting bd56f2b3042f4d45960ac35b45e502be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682229243389 2023-04-23 05:54:06,172 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting d7946e18f3ab4d3eb5f2e93850115c3b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682229243389 2023-04-23 05:54:06,172 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 09da67adfdf2489ca821de62846d5afc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682229244006 2023-04-23 05:54:06,172 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2b52745d66c64757acc9928db7bec9ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682229245139 2023-04-23 05:54:06,173 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting cab7221bafe04691b73717ec92638932, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682229244006 2023-04-23 05:54:06,173 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 65f073c9dc55499f9802d015da5ad735, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682229245139 2023-04-23 05:54:06,195 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#B#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:06,206 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#A#compaction#79 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:06,229 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/ab34329bb50e4776a270ec17bd31dc64 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/ab34329bb50e4776a270ec17bd31dc64 2023-04-23 05:54:06,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:06,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:06,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:06,236 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/A of bfd1e7528fb104543b2ee708b991f02a into ab34329bb50e4776a270ec17bd31dc64(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:06,236 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:06,236 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/A, priority=12, startTime=1682229246167; duration=0sec 2023-04-23 05:54:06,236 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:06,236 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:54:06,238 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:54:06,238 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/C is initiating minor compaction (all files) 2023-04-23 05:54:06,238 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/C in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:06,239 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5ed8e65a805747319ce667f7c57fd9bb, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/87e69ff16391460c9e85f63de48b21f6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/cc14f31466a843259f83bc32da29077c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1af43ab28441444c9b044817c8d5d447] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=48.3 K 2023-04-23 05:54:06,239 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 5ed8e65a805747319ce667f7c57fd9bb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682229243290 2023-04-23 05:54:06,239 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 87e69ff16391460c9e85f63de48b21f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682229243389 2023-04-23 05:54:06,240 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting cc14f31466a843259f83bc32da29077c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682229244006 2023-04-23 05:54:06,240 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 1af43ab28441444c9b044817c8d5d447, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682229245139 2023-04-23 05:54:06,250 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#C#compaction#80 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:06,261 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/1040f37feb934a35ba9aee9d6d00226b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1040f37feb934a35ba9aee9d6d00226b 2023-04-23 05:54:06,270 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/C of bfd1e7528fb104543b2ee708b991f02a into 1040f37feb934a35ba9aee9d6d00226b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:06,270 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:06,270 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/C, priority=12, startTime=1682229246167; duration=0sec 2023-04-23 05:54:06,270 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:06,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:06,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:06,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:06,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:06,277 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:06,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:06,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:06,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:06,277 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:06,315 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,315 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,315 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:06,316 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,347 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:06,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:06,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:06,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:06,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:06,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:06,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:06,347 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:06,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:06,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,347 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:06,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,348 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:06,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:06,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:06,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-23 05:54:06,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:06,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:06,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:06,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:06,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:06,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:06,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/24ab69b2209a4263af7ac278fa47f9b8 2023-04-23 05:54:06,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:06,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229306426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:06,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229306437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,448 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:06,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:06,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:06,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:06,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:06,469 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:06,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:06,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:06,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:06,469 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/902d0411a9e5485fb018f3dd8d5a3243 2023-04-23 05:54:06,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:06,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229306529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:06,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229306539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:06,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:06,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:06,574 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,574 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:06,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:06,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:06,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:06,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:06,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:06,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:06,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:06,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:06,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:06,576 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:06,576 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:06,576 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:06,576 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:06,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:06,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:06,577 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,577 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:06,577 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:06,615 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:06,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:06,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:06,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:06,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:06,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b8d2fac[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:06,617 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:06,617 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,632 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/95c876b197ab4c6ea0649180cd7870bd as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/95c876b197ab4c6ea0649180cd7870bd 2023-04-23 05:54:06,640 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/B of bfd1e7528fb104543b2ee708b991f02a into 95c876b197ab4c6ea0649180cd7870bd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:06,640 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:06,640 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/B, priority=12, startTime=1682229246167; duration=0sec 2023-04-23 05:54:06,640 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:06,654 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,654 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:06,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:06,654 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:06,654 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:06,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:06,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:06,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:06,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:06,720 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:06,720 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:06,720 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:06,720 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:06,720 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:06,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:06,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229306731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:06,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229306745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,787 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,787 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:06,788 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:06,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:06,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:06,789 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:06,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:06,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:06,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:06,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:06,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:06,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,862 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:06,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:06,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:06,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:06,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:06,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:06,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,863 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:06,863 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:06,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:06,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:06,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:06,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/c6e4362310e341cb81d07ab4fbcd69cd 2023-04-23 05:54:06,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/24ab69b2209a4263af7ac278fa47f9b8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/24ab69b2209a4263af7ac278fa47f9b8 2023-04-23 05:54:06,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/24ab69b2209a4263af7ac278fa47f9b8, entries=150, sequenceid=470, filesize=11.9 K 2023-04-23 05:54:06,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/902d0411a9e5485fb018f3dd8d5a3243 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/902d0411a9e5485fb018f3dd8d5a3243 2023-04-23 05:54:06,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/902d0411a9e5485fb018f3dd8d5a3243, entries=150, sequenceid=470, filesize=11.9 K 2023-04-23 05:54:06,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/c6e4362310e341cb81d07ab4fbcd69cd as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c6e4362310e341cb81d07ab4fbcd69cd 2023-04-23 05:54:06,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c6e4362310e341cb81d07ab4fbcd69cd, entries=150, sequenceid=470, filesize=11.9 K 2023-04-23 05:54:06,947 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bfd1e7528fb104543b2ee708b991f02a in 539ms, sequenceid=470, compaction requested=false 2023-04-23 05:54:06,947 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:06,963 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:06,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:06,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:06,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:06,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:06,972 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:06,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:06,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:06,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:06,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:06,973 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:07,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:07,013 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,013 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:07,013 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:54:07,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:07,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:07,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:07,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:07,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:07,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:07,024 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/eb98ac79b14e4c8186c4b378bfd86d54 2023-04-23 05:54:07,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:07,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:07,056 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/9acf6b9bd6d04104bfd739962e04c34a 2023-04-23 05:54:07,082 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/d91918dee1b44f87832e38088c40bcde 2023-04-23 05:54:07,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229247062, max timestamp=1682229247081Num uniques 859; 2023-04-23 05:54:07,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:07,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229247062, max timestamp=1682229247081Num uniques 859; 2023-04-23 05:54:07,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:07,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229247062, max timestamp=1682229247081Num uniques 859; 2023-04-23 05:54:07,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:07,090 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/eb98ac79b14e4c8186c4b378bfd86d54 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/eb98ac79b14e4c8186c4b378bfd86d54 2023-04-23 05:54:07,097 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/eb98ac79b14e4c8186c4b378bfd86d54, entries=150, sequenceid=483, filesize=11.9 K 2023-04-23 05:54:07,098 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/9acf6b9bd6d04104bfd739962e04c34a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9acf6b9bd6d04104bfd739962e04c34a 2023-04-23 05:54:07,111 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9acf6b9bd6d04104bfd739962e04c34a, entries=150, sequenceid=483, filesize=11.9 K 2023-04-23 05:54:07,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/d91918dee1b44f87832e38088c40bcde as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d91918dee1b44f87832e38088c40bcde 2023-04-23 05:54:07,123 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d91918dee1b44f87832e38088c40bcde, entries=150, sequenceid=483, filesize=11.9 K 2023-04-23 05:54:07,124 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=100.63 KB/103050 for bfd1e7528fb104543b2ee708b991f02a in 111ms, sequenceid=483, compaction requested=true 2023-04-23 05:54:07,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:07,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,124 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:07,125 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:07,125 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:07,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:07,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=107.34 KB heapSize=305.30 KB 2023-04-23 05:54:07,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:07,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:07,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:07,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:07,134 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:07,134 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:07,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229307164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,173 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/3ac0ee1cc180485591258215e1ee9513 2023-04-23 05:54:07,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229307171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,192 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,193 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,193 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:07,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb9287f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:07,197 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:07,197 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,207 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/5c2c1ce3f8454840a72f2f1892bbd529 2023-04-23 05:54:07,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:07,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:07,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:07,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:07,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/8af12f3f9ff447f59c9c8d3429dfe227 2023-04-23 05:54:07,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/3ac0ee1cc180485591258215e1ee9513 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3ac0ee1cc180485591258215e1ee9513 2023-04-23 05:54:07,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3ac0ee1cc180485591258215e1ee9513, entries=150, sequenceid=516, filesize=11.9 K 2023-04-23 05:54:07,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/5c2c1ce3f8454840a72f2f1892bbd529 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/5c2c1ce3f8454840a72f2f1892bbd529 2023-04-23 05:54:07,264 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/5c2c1ce3f8454840a72f2f1892bbd529, entries=150, sequenceid=516, filesize=11.9 K 2023-04-23 05:54:07,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/8af12f3f9ff447f59c9c8d3429dfe227 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/8af12f3f9ff447f59c9c8d3429dfe227 2023-04-23 05:54:07,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229307273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229307277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/8af12f3f9ff447f59c9c8d3429dfe227, entries=150, sequenceid=516, filesize=11.9 K 2023-04-23 05:54:07,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~305.25 KB/312576, currentSize=73.80 KB/75570 for bfd1e7528fb104543b2ee708b991f02a in 149ms, sequenceid=516, compaction requested=true 2023-04-23 05:54:07,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:07,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:07,282 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:54:07,282 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:54:07,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:07,282 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:07,292 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:54:07,292 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:54:07,292 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/A is initiating minor compaction (all files) 2023-04-23 05:54:07,293 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/B is initiating minor compaction (all files) 2023-04-23 05:54:07,293 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/A in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,293 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/B in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,293 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/ab34329bb50e4776a270ec17bd31dc64, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/24ab69b2209a4263af7ac278fa47f9b8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/eb98ac79b14e4c8186c4b378bfd86d54, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3ac0ee1cc180485591258215e1ee9513] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=48.5 K 2023-04-23 05:54:07,293 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/95c876b197ab4c6ea0649180cd7870bd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/902d0411a9e5485fb018f3dd8d5a3243, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9acf6b9bd6d04104bfd739962e04c34a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/5c2c1ce3f8454840a72f2f1892bbd529] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=48.5 K 2023-04-23 05:54:07,294 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 95c876b197ab4c6ea0649180cd7870bd, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682229245139 2023-04-23 05:54:07,294 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting ab34329bb50e4776a270ec17bd31dc64, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682229245139 2023-04-23 05:54:07,294 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 24ab69b2209a4263af7ac278fa47f9b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1682229245289 2023-04-23 05:54:07,294 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 902d0411a9e5485fb018f3dd8d5a3243, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1682229245289 2023-04-23 05:54:07,296 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting eb98ac79b14e4c8186c4b378bfd86d54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1682229246420 2023-04-23 05:54:07,297 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9acf6b9bd6d04104bfd739962e04c34a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1682229246420 2023-04-23 05:54:07,297 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 3ac0ee1cc180485591258215e1ee9513, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682229247102 2023-04-23 05:54:07,297 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 5c2c1ce3f8454840a72f2f1892bbd529, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682229247102 2023-04-23 05:54:07,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:07,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:07,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:07,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:07,304 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:07,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:07,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:07,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:07,304 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:07,322 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#A#compaction#90 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:07,325 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#B#compaction#91 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:07,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,362 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,364 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,376 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/d357098564684606978529421a25d096 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/d357098564684606978529421a25d096 2023-04-23 05:54:07,383 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/B of bfd1e7528fb104543b2ee708b991f02a into d357098564684606978529421a25d096(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:07,383 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:07,383 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/B, priority=12, startTime=1682229247282; duration=0sec 2023-04-23 05:54:07,383 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:07,384 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:54:07,387 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:54:07,387 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): bfd1e7528fb104543b2ee708b991f02a/C is initiating minor compaction (all files) 2023-04-23 05:54:07,387 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bfd1e7528fb104543b2ee708b991f02a/C in TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,388 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1040f37feb934a35ba9aee9d6d00226b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c6e4362310e341cb81d07ab4fbcd69cd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d91918dee1b44f87832e38088c40bcde, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/8af12f3f9ff447f59c9c8d3429dfe227] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp, totalSize=48.5 K 2023-04-23 05:54:07,388 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1040f37feb934a35ba9aee9d6d00226b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1682229245139 2023-04-23 05:54:07,389 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c6e4362310e341cb81d07ab4fbcd69cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1682229245289 2023-04-23 05:54:07,389 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d91918dee1b44f87832e38088c40bcde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=483, earliestPutTs=1682229246420 2023-04-23 05:54:07,389 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 8af12f3f9ff447f59c9c8d3429dfe227, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1682229247102 2023-04-23 05:54:07,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:07,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:07,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,392 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:07,393 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:07,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:07,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:07,401 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): bfd1e7528fb104543b2ee708b991f02a#C#compaction#92 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:07,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:07,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-23 05:54:07,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:07,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:07,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:07,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:07,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:07,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:07,493 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:07,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:07,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:07,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:07,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:07,500 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:07,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:07,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:07,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:07,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,503 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:07,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:07,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:07,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:07,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:07,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:07,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:07,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:07,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:07,507 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,507 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:07,507 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:07,507 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:07,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:07,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:07,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:07,511 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a3cd4f8ea6ed4f4f8e26cb99daafdd59 2023-04-23 05:54:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229307513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:07,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@778522cb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:07,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:07,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229307515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:07,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:07,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:07,519 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,519 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:07,524 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:07,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:07,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:07,530 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:07,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:07,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:07,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:07,530 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:07,532 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,532 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,533 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:07,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,535 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,537 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:07,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:07,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:07,537 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:07,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:07,541 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,541 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:07,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56592 deadline: 1682229307540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,554 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:54:07,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/719359dc044c42418dfa53157c977b21 2023-04-23 05:54:07,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229307617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229307617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,637 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:07,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:07,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:07,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:07,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:07,644 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:07,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:07,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:07,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:07,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,646 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:07,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:07,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:07,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:07,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:07,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:07,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:07,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:07,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:07,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:07,648 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,648 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:07,649 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:07,649 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:07,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:07,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:07,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:07,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:07,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:07,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:07,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@649e026c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:07,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,659 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:07,659 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:07,659 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:07,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:07,661 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:07,661 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,661 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,661 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:07,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:07,666 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:07,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:07,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:07,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:07,667 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:07,669 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,669 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,669 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:07,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,671 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:07,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:07,675 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:07,676 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:07,676 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:07,765 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/47d99ec6253f4dbbb335270a796c5f00 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/47d99ec6253f4dbbb335270a796c5f00 2023-04-23 05:54:07,774 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/A of bfd1e7528fb104543b2ee708b991f02a into 47d99ec6253f4dbbb335270a796c5f00(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:07,774 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:07,774 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/A, priority=12, startTime=1682229247282; duration=0sec 2023-04-23 05:54:07,774 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:07,776 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:07,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:07,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:07,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:07,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:07,781 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:07,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:07,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:07,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:07,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,784 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,784 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,784 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:07,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:07,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:07,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:07,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:07,787 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:07,787 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:07,787 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:07,787 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:07,787 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:07,787 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,787 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:07,787 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:07,788 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:07,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:07,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:07,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:07,792 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,792 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,792 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,792 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:07,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:07,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66eca11c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:07,795 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:07,795 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,797 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,797 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:07,797 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:07,797 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:07,798 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,798 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:07,799 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:07,799 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,799 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,799 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:07,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:07,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:07,803 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:07,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:07,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:07,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:07,803 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:07,804 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,804 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,804 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,808 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,810 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:07,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:07,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,811 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:07,811 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:07,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:07,825 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/8f8a69f61e914110be87d85a2e35daee as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/8f8a69f61e914110be87d85a2e35daee 2023-04-23 05:54:07,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229307823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:07,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229307826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,834 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bfd1e7528fb104543b2ee708b991f02a/C of bfd1e7528fb104543b2ee708b991f02a into 8f8a69f61e914110be87d85a2e35daee(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:07,834 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:07,834 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., storeName=bfd1e7528fb104543b2ee708b991f02a/C, priority=12, startTime=1682229247282; duration=0sec 2023-04-23 05:54:07,835 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:07,911 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:07,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:07,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:07,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:07,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:07,914 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:07,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:07,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:07,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:07,915 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,915 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,917 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,918 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,918 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:07,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:07,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:07,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:07,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:07,919 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:07,919 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:07,919 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:07,920 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,920 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:07,920 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:07,920 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:07,920 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:07,921 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,922 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,922 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,922 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:07,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44b4876e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:07,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:07,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,925 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,925 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:07,925 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:07,925 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:07,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:07,926 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,926 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,926 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,927 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:07,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:07,929 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:07,929 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:07,929 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:07,929 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:07,929 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:07,932 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,932 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,932 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:07,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:07,933 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:07,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:07,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:07,937 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:07,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:07,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:07,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:07,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:07,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:07,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:07,938 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:07,938 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:07,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:07,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:07,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:08,012 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/53e16ee567864a7f979a43881f077d80 2023-04-23 05:54:08,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a3cd4f8ea6ed4f4f8e26cb99daafdd59 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3cd4f8ea6ed4f4f8e26cb99daafdd59 2023-04-23 05:54:08,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3cd4f8ea6ed4f4f8e26cb99daafdd59, entries=250, sequenceid=533, filesize=16.7 K 2023-04-23 05:54:08,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/719359dc044c42418dfa53157c977b21 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/719359dc044c42418dfa53157c977b21 2023-04-23 05:54:08,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/719359dc044c42418dfa53157c977b21, entries=150, sequenceid=533, filesize=11.9 K 2023-04-23 05:54:08,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/53e16ee567864a7f979a43881f077d80 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/53e16ee567864a7f979a43881f077d80 2023-04-23 05:54:08,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/53e16ee567864a7f979a43881f077d80, entries=150, sequenceid=533, filesize=11.9 K 2023-04-23 05:54:08,038 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:08,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:08,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bfd1e7528fb104543b2ee708b991f02a in 557ms, sequenceid=533, compaction requested=false 2023-04-23 05:54:08,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:08,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:08,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:08,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:08,044 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:08,044 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:08,044 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:08,044 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:08,044 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:08,044 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:08,046 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:08,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:08,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:08,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:08,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:08,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:08,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:08,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:08,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:08,047 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:08,047 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:08,047 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:08,048 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:08,048 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:08,048 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:08,048 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:08,048 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-23 05:54:08,048 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:08,048 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:08,049 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:08,049 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:08,049 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:08,049 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:08,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:08,129 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. as already flushing 2023-04-23 05:54:08,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229308153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229308154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229308255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229308255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,458 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a9a79a2d80e4424fb3e32d1ad2baca4c 2023-04-23 05:54:08,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229308459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229308460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229308762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229308763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56626 deadline: 1682229308771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,774 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18313 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:54:08,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:08,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56652 deadline: 1682229308854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:08,858 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18400 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:54:08,874 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/579b735d7dd14cdf9ee3694edd66dafc 2023-04-23 05:54:09,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:09,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56582 deadline: 1682229309267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:09,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56650 deadline: 1682229309268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,291 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/6d398b82795047d3b4a19786c8fe0c06 2023-04-23 05:54:09,297 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a9a79a2d80e4424fb3e32d1ad2baca4c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a9a79a2d80e4424fb3e32d1ad2baca4c 2023-04-23 05:54:09,301 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a9a79a2d80e4424fb3e32d1ad2baca4c, entries=150, sequenceid=551, filesize=11.9 K 2023-04-23 05:54:09,302 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/579b735d7dd14cdf9ee3694edd66dafc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/579b735d7dd14cdf9ee3694edd66dafc 2023-04-23 05:54:09,307 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/579b735d7dd14cdf9ee3694edd66dafc, entries=150, sequenceid=551, filesize=11.9 K 2023-04-23 05:54:09,308 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/6d398b82795047d3b4a19786c8fe0c06 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/6d398b82795047d3b4a19786c8fe0c06 2023-04-23 05:54:09,312 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/6d398b82795047d3b4a19786c8fe0c06, entries=150, sequenceid=551, filesize=11.9 K 2023-04-23 05:54:09,313 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for bfd1e7528fb104543b2ee708b991f02a in 1265ms, sequenceid=551, compaction requested=true 2023-04-23 05:54:09,313 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:09,313 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:09,313 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:09,314 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:09,315 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:09,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:09,316 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,316 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:09,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:09,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1af97649[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:09,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:09,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,320 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:09,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:09,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:09,320 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,320 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:09,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,331 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:09,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:09,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:09,331 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:09,331 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:09,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:09,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:09,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:09,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:09,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:09,336 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:09,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:09,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:09,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:09,336 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:09,338 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,338 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:09,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,338 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:09,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:09,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:09,339 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:09,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:09,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,343 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:09,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:09,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:09,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:09,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,344 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:09,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:09,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:09,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:09,344 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:09,344 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:09,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:09,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:09,444 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:09,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:09,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:09,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:09,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:09,448 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:09,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:09,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:09,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:09,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:09,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:09,451 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:09,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:09,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:09,452 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,452 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:09,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:09,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:09,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:09,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:09,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:09,453 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:09,453 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:09,453 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:09,453 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:09,453 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:09,453 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:09,453 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:09,454 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-23 05:54:09,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:09,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:09,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:09,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:09,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:09,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:09,865 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a55bcd916f484c4dbb5fb40e499ec2af 2023-04-23 05:54:09,882 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/470d461500c54c6f9321555de49f582c 2023-04-23 05:54:09,896 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/66ce4efa4c7f46e99e60db3a8a464fe3 2023-04-23 05:54:09,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/a55bcd916f484c4dbb5fb40e499ec2af as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a55bcd916f484c4dbb5fb40e499ec2af 2023-04-23 05:54:09,908 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a55bcd916f484c4dbb5fb40e499ec2af, entries=150, sequenceid=568, filesize=11.9 K 2023-04-23 05:54:09,909 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/470d461500c54c6f9321555de49f582c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/470d461500c54c6f9321555de49f582c 2023-04-23 05:54:09,915 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/470d461500c54c6f9321555de49f582c, entries=150, sequenceid=568, filesize=11.9 K 2023-04-23 05:54:09,916 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/66ce4efa4c7f46e99e60db3a8a464fe3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/66ce4efa4c7f46e99e60db3a8a464fe3 2023-04-23 05:54:09,922 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/66ce4efa4c7f46e99e60db3a8a464fe3, entries=150, sequenceid=568, filesize=11.9 K 2023-04-23 05:54:09,923 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for bfd1e7528fb104543b2ee708b991f02a in 469ms, sequenceid=568, compaction requested=true 2023-04-23 05:54:09,923 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:09,923 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:09,923 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:09,923 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:09,923 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,923 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:09,923 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:09,925 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,925 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:09,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:09,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,926 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:09,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:09,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:09,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e609a18[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:09,927 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:09,927 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,929 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,929 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,929 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:09,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:09,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:09,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:09,930 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,930 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:09,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:09,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:09,930 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:09,931 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:09,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:09,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:09,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:09,933 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:09,934 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:09,934 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:09,934 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:09,934 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:09,936 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:09,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:09,937 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:09,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,939 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:09,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,941 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,941 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:09,941 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:09,941 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:09,941 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:09,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:09,941 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:09,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:09,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:09,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:09,942 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:09,942 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:09,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:09,941 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,942 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:09,942 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:09,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:09,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:10,042 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:10,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:10,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:10,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:10,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:10,046 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:10,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:10,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:10,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:10,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:10,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:10,047 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:10,047 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:10,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:10,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:10,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:10,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:10,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:10,049 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:10,049 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. started... 2023-04-23 05:54:10,050 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:10,050 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:10,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:10,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,051 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:10,052 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:10,052 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:10,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:10,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d873e89[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:10,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:10,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:10,059 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:10,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:10,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:10,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:10,059 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,059 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:10,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:10,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:10,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:10,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:10,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:10,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:10,063 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:10,063 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:10,063 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:10,064 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:10,064 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:10,065 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:10,065 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:10,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:10,065 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:10,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:10,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:10,066 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:10,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:10,067 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:10,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:10,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:10,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:10,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:10,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:10,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:10,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:10,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:10,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:10,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:10,069 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:10,069 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:10,070 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:10,070 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:10,070 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:10,070 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:10,070 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:10,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:10,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:10,144 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x142b9819 to 127.0.0.1:63802 2023-04-23 05:54:10,144 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:10,145 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbe28d to 127.0.0.1:63802 2023-04-23 05:54:10,145 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:10,145 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64804459 to 127.0.0.1:63802 2023-04-23 05:54:10,145 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:10,150 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c9ded19 to 127.0.0.1:63802 2023-04-23 05:54:10,151 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:10,170 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:10,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:10,272 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36dc7f7f to 127.0.0.1:63802 2023-04-23 05:54:10,272 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:10,274 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75f5a5e6 to 127.0.0.1:63802 2023-04-23 05:54:10,274 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:11,573 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e82daee to 127.0.0.1:63802 2023-04-23 05:54:11,573 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:18,870 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79fee867 to 127.0.0.1:63802 2023-04-23 05:54:18,870 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:18,939 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e2005d0 to 127.0.0.1:63802 2023-04-23 05:54:18,939 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:18,939 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 128 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 173 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 6 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 7 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 156 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 4180 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 4293 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 1841 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 5523 rows 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 1826 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 5473 rows 2023-04-23 05:54:18,940 INFO [Listener at localhost/41845] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-23 05:54:18,940 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23162110 to 127.0.0.1:63802 2023-04-23 05:54:18,940 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:18,945 INFO [Listener at localhost/41845] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-23 05:54:18,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-23 05:54:18,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:18,959 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229258959"}]},"ts":"1682229258959"} 2023-04-23 05:54:18,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-23 05:54:18,960 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-23 05:54:18,962 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-23 05:54:18,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfd1e7528fb104543b2ee708b991f02a, UNASSIGN}] 2023-04-23 05:54:18,964 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfd1e7528fb104543b2ee708b991f02a, UNASSIGN 2023-04-23 05:54:18,965 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=bfd1e7528fb104543b2ee708b991f02a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:18,965 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229258965"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229258965"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229258965"}]},"ts":"1682229258965"} 2023-04-23 05:54:18,966 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:54:19,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-23 05:54:19,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:19,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bfd1e7528fb104543b2ee708b991f02a, disabling compactions & flushes 2023-04-23 05:54:19,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:19,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:19,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. after waiting 0 ms 2023-04-23 05:54:19,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:19,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing bfd1e7528fb104543b2ee708b991f02a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-23 05:54:19,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=A 2023-04-23 05:54:19,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:19,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=B 2023-04-23 05:54:19,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:19,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bfd1e7528fb104543b2ee708b991f02a, store=C 2023-04-23 05:54:19,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:19,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-23 05:54:19,535 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/41d58d6cd5164f3f9278182f74e54c61 2023-04-23 05:54:19,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-23 05:54:19,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/c7167a3f1bb247028ff6c28ef2af87c8 2023-04-23 05:54:20,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-23 05:54:20,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/d33504bcd42646d8a9cbe92b709f4280 2023-04-23 05:54:20,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/A/41d58d6cd5164f3f9278182f74e54c61 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/41d58d6cd5164f3f9278182f74e54c61 2023-04-23 05:54:20,367 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/41d58d6cd5164f3f9278182f74e54c61, entries=150, sequenceid=577, filesize=11.9 K 2023-04-23 05:54:20,368 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/B/c7167a3f1bb247028ff6c28ef2af87c8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c7167a3f1bb247028ff6c28ef2af87c8 2023-04-23 05:54:20,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c7167a3f1bb247028ff6c28ef2af87c8, entries=150, sequenceid=577, filesize=11.9 K 2023-04-23 05:54:20,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/.tmp/C/d33504bcd42646d8a9cbe92b709f4280 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d33504bcd42646d8a9cbe92b709f4280 2023-04-23 05:54:20,377 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d33504bcd42646d8a9cbe92b709f4280, entries=150, sequenceid=577, filesize=11.9 K 2023-04-23 05:54:20,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for bfd1e7528fb104543b2ee708b991f02a in 1253ms, sequenceid=577, compaction requested=true 2023-04-23 05:54:20,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4aaa4a03c43a4c2da3e6a21cb6ede65c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3d79f23b0a6421f9c71501906fe2892, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3f478cad4bd442ce8fc7ca0ed0629c21, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/f191e8d6804f4bf793bffddb08053351, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0160d4b395cd41368de3e39389e2b9dd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0a8b75b0b7624aa1919f8a120e19192e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/2d982d00b5b24c6da99d8aae1069adff, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/932c18fb02644d07b406c1b743701eb7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/8ed391f0c7ee492c94cbe5a2a804067b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/723e2b0c5c1a48c6bebed84398e7eb05, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/69a21fbb7bad401ab1b9d14a32a98313, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/e7d210f892df4ee0815909adf9dca9ce, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5f7938b5ac3f47c99df255d9f07269f0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/94b543d6ed98418bb2d0dddf1f9169e5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/03e447eaad194a79ad27f6b22f03b1a3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5a8f199c37e344d7abdfe7b3b19a7394, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d77d5a12f6ed41c6a81142735247579b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cf1495f044f4404685b749b5f71b4bf1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/deae3002a4644ac9914c543f091e0bac, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/268145bea74f48148f92b4f076c18b91, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4214df568174487188b4952d49fbced2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a030b4dac16c4f158ad00308acbd0d1d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4a8d33552f134d0b9e8ad3e7a7e6a8c1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d7946e18f3ab4d3eb5f2e93850115c3b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cab7221bafe04691b73717ec92638932, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/65f073c9dc55499f9802d015da5ad735, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/ab34329bb50e4776a270ec17bd31dc64, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/24ab69b2209a4263af7ac278fa47f9b8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/eb98ac79b14e4c8186c4b378bfd86d54, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3ac0ee1cc180485591258215e1ee9513] to archive 2023-04-23 05:54:20,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:54:20,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4aaa4a03c43a4c2da3e6a21cb6ede65c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4aaa4a03c43a4c2da3e6a21cb6ede65c 2023-04-23 05:54:20,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3d79f23b0a6421f9c71501906fe2892 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3d79f23b0a6421f9c71501906fe2892 2023-04-23 05:54:20,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3f478cad4bd442ce8fc7ca0ed0629c21 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3f478cad4bd442ce8fc7ca0ed0629c21 2023-04-23 05:54:20,406 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/f191e8d6804f4bf793bffddb08053351 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/f191e8d6804f4bf793bffddb08053351 2023-04-23 05:54:20,407 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0160d4b395cd41368de3e39389e2b9dd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0160d4b395cd41368de3e39389e2b9dd 2023-04-23 05:54:20,408 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0a8b75b0b7624aa1919f8a120e19192e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/0a8b75b0b7624aa1919f8a120e19192e 2023-04-23 05:54:20,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/2d982d00b5b24c6da99d8aae1069adff to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/2d982d00b5b24c6da99d8aae1069adff 2023-04-23 05:54:20,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/932c18fb02644d07b406c1b743701eb7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/932c18fb02644d07b406c1b743701eb7 2023-04-23 05:54:20,412 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/8ed391f0c7ee492c94cbe5a2a804067b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/8ed391f0c7ee492c94cbe5a2a804067b 2023-04-23 05:54:20,413 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/723e2b0c5c1a48c6bebed84398e7eb05 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/723e2b0c5c1a48c6bebed84398e7eb05 2023-04-23 05:54:20,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/69a21fbb7bad401ab1b9d14a32a98313 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/69a21fbb7bad401ab1b9d14a32a98313 2023-04-23 05:54:20,415 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/e7d210f892df4ee0815909adf9dca9ce to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/e7d210f892df4ee0815909adf9dca9ce 2023-04-23 05:54:20,416 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5f7938b5ac3f47c99df255d9f07269f0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5f7938b5ac3f47c99df255d9f07269f0 2023-04-23 05:54:20,418 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/94b543d6ed98418bb2d0dddf1f9169e5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/94b543d6ed98418bb2d0dddf1f9169e5 2023-04-23 05:54:20,419 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/03e447eaad194a79ad27f6b22f03b1a3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/03e447eaad194a79ad27f6b22f03b1a3 2023-04-23 05:54:20,421 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5a8f199c37e344d7abdfe7b3b19a7394 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/5a8f199c37e344d7abdfe7b3b19a7394 2023-04-23 05:54:20,422 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d77d5a12f6ed41c6a81142735247579b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d77d5a12f6ed41c6a81142735247579b 2023-04-23 05:54:20,424 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cf1495f044f4404685b749b5f71b4bf1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cf1495f044f4404685b749b5f71b4bf1 2023-04-23 05:54:20,425 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/deae3002a4644ac9914c543f091e0bac to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/deae3002a4644ac9914c543f091e0bac 2023-04-23 05:54:20,426 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/268145bea74f48148f92b4f076c18b91 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/268145bea74f48148f92b4f076c18b91 2023-04-23 05:54:20,427 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4214df568174487188b4952d49fbced2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4214df568174487188b4952d49fbced2 2023-04-23 05:54:20,428 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a030b4dac16c4f158ad00308acbd0d1d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a030b4dac16c4f158ad00308acbd0d1d 2023-04-23 05:54:20,429 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4a8d33552f134d0b9e8ad3e7a7e6a8c1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/4a8d33552f134d0b9e8ad3e7a7e6a8c1 2023-04-23 05:54:20,430 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d7946e18f3ab4d3eb5f2e93850115c3b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/d7946e18f3ab4d3eb5f2e93850115c3b 2023-04-23 05:54:20,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cab7221bafe04691b73717ec92638932 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/cab7221bafe04691b73717ec92638932 2023-04-23 05:54:20,432 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/65f073c9dc55499f9802d015da5ad735 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/65f073c9dc55499f9802d015da5ad735 2023-04-23 05:54:20,433 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/ab34329bb50e4776a270ec17bd31dc64 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/ab34329bb50e4776a270ec17bd31dc64 2023-04-23 05:54:20,435 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/24ab69b2209a4263af7ac278fa47f9b8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/24ab69b2209a4263af7ac278fa47f9b8 2023-04-23 05:54:20,436 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/eb98ac79b14e4c8186c4b378bfd86d54 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/eb98ac79b14e4c8186c4b378bfd86d54 2023-04-23 05:54:20,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3ac0ee1cc180485591258215e1ee9513 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/3ac0ee1cc180485591258215e1ee9513 2023-04-23 05:54:20,461 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/fdb8e3e24bf74e67b0fb1df448b84fcf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/27eb1a047c5f46698ef42436f69b4411, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/d959a4583d374507bb415b28ddf8d829, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/7a4755c024234bb2831fd759d93dbee7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b0ea5bedc5e24942b78b12e3615bf163, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/22bddd084cb24915808eae9ae041e44f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/cebb8893d7ab4ca5930cafa06f84e027, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a16cbb31c1dc4496bf56c0978e4a77a1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/47f7f431ec9a4f24868a43b6aab7cea1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/aabd230b48464c2fa6a7b5b3554cbf4e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c09a077f111842b2b19fb2bc82c26c80, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/35f032e7857a4548aefadde93266984f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/dc104c27820c45f783ea087c5b52887d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a47edd96442144d1a62a46f4d3689c53, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/832c8fff0d674172851b1e486ea1c6a4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/3c71756928384aa08621dfc35354675d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b383fd28dc2741349ab7a3fdf82b63b2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/83046b576e4844198d84fb2e4f1c94a6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c63c227c182f4666a0eaf2bb4f4a250e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/4d6b6ac15b6d43d38683d84365c7e43f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/ecac09eeb0b34ec3a08c4893a4567b2c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/e49ab6c1c6c645e69b99c192aa37faaf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9e276a45e91b407bafb26da64ee84c8f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/bd56f2b3042f4d45960ac35b45e502be, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/09da67adfdf2489ca821de62846d5afc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/95c876b197ab4c6ea0649180cd7870bd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/2b52745d66c64757acc9928db7bec9ea, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/902d0411a9e5485fb018f3dd8d5a3243, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9acf6b9bd6d04104bfd739962e04c34a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/5c2c1ce3f8454840a72f2f1892bbd529] to archive 2023-04-23 05:54:20,462 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:54:20,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/fdb8e3e24bf74e67b0fb1df448b84fcf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/fdb8e3e24bf74e67b0fb1df448b84fcf 2023-04-23 05:54:20,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/27eb1a047c5f46698ef42436f69b4411 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/27eb1a047c5f46698ef42436f69b4411 2023-04-23 05:54:20,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/d959a4583d374507bb415b28ddf8d829 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/d959a4583d374507bb415b28ddf8d829 2023-04-23 05:54:20,468 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/7a4755c024234bb2831fd759d93dbee7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/7a4755c024234bb2831fd759d93dbee7 2023-04-23 05:54:20,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b0ea5bedc5e24942b78b12e3615bf163 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b0ea5bedc5e24942b78b12e3615bf163 2023-04-23 05:54:20,470 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/22bddd084cb24915808eae9ae041e44f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/22bddd084cb24915808eae9ae041e44f 2023-04-23 05:54:20,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/cebb8893d7ab4ca5930cafa06f84e027 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/cebb8893d7ab4ca5930cafa06f84e027 2023-04-23 05:54:20,472 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a16cbb31c1dc4496bf56c0978e4a77a1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a16cbb31c1dc4496bf56c0978e4a77a1 2023-04-23 05:54:20,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/47f7f431ec9a4f24868a43b6aab7cea1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/47f7f431ec9a4f24868a43b6aab7cea1 2023-04-23 05:54:20,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/aabd230b48464c2fa6a7b5b3554cbf4e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/aabd230b48464c2fa6a7b5b3554cbf4e 2023-04-23 05:54:20,475 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c09a077f111842b2b19fb2bc82c26c80 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c09a077f111842b2b19fb2bc82c26c80 2023-04-23 05:54:20,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/35f032e7857a4548aefadde93266984f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/35f032e7857a4548aefadde93266984f 2023-04-23 05:54:20,477 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/dc104c27820c45f783ea087c5b52887d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/dc104c27820c45f783ea087c5b52887d 2023-04-23 05:54:20,478 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a47edd96442144d1a62a46f4d3689c53 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/a47edd96442144d1a62a46f4d3689c53 2023-04-23 05:54:20,479 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/832c8fff0d674172851b1e486ea1c6a4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/832c8fff0d674172851b1e486ea1c6a4 2023-04-23 05:54:20,480 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/3c71756928384aa08621dfc35354675d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/3c71756928384aa08621dfc35354675d 2023-04-23 05:54:20,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b383fd28dc2741349ab7a3fdf82b63b2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/b383fd28dc2741349ab7a3fdf82b63b2 2023-04-23 05:54:20,482 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/83046b576e4844198d84fb2e4f1c94a6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/83046b576e4844198d84fb2e4f1c94a6 2023-04-23 05:54:20,483 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c63c227c182f4666a0eaf2bb4f4a250e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c63c227c182f4666a0eaf2bb4f4a250e 2023-04-23 05:54:20,484 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/4d6b6ac15b6d43d38683d84365c7e43f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/4d6b6ac15b6d43d38683d84365c7e43f 2023-04-23 05:54:20,486 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/ecac09eeb0b34ec3a08c4893a4567b2c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/ecac09eeb0b34ec3a08c4893a4567b2c 2023-04-23 05:54:20,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/e49ab6c1c6c645e69b99c192aa37faaf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/e49ab6c1c6c645e69b99c192aa37faaf 2023-04-23 05:54:20,488 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9e276a45e91b407bafb26da64ee84c8f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9e276a45e91b407bafb26da64ee84c8f 2023-04-23 05:54:20,489 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/bd56f2b3042f4d45960ac35b45e502be to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/bd56f2b3042f4d45960ac35b45e502be 2023-04-23 05:54:20,490 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/09da67adfdf2489ca821de62846d5afc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/09da67adfdf2489ca821de62846d5afc 2023-04-23 05:54:20,491 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/95c876b197ab4c6ea0649180cd7870bd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/95c876b197ab4c6ea0649180cd7870bd 2023-04-23 05:54:20,492 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/2b52745d66c64757acc9928db7bec9ea to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/2b52745d66c64757acc9928db7bec9ea 2023-04-23 05:54:20,493 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/902d0411a9e5485fb018f3dd8d5a3243 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/902d0411a9e5485fb018f3dd8d5a3243 2023-04-23 05:54:20,494 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9acf6b9bd6d04104bfd739962e04c34a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/9acf6b9bd6d04104bfd739962e04c34a 2023-04-23 05:54:20,495 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/5c2c1ce3f8454840a72f2f1892bbd529 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/5c2c1ce3f8454840a72f2f1892bbd529 2023-04-23 05:54:20,499 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a49050e3142c4a1c816f48361793bac8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d96be4f3255c497cbee34c3023ee22e3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ff0e9490797b483aa9231e9c9e558dc6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/149003e3dc6c4b2993336f2dbeb9d249, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5809d075b4f642038cd20d0cff85c082, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c7439bec5fc04786a7f48781df577aa6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/beaaddb75dde46518d79814f03a36d47, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/f4db1da8c66347bb8e1b7ee40052a805, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a14e4fb37eb44109a342781c3ec49cb4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/b412b4919e9f4f13b0d1aa27dd0a0f36, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ddddc13f5b5a40dfa4b3ce50f6d308c1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a1919cdbc76e4028ab7bd6538902627f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/aabcf50ab17a4896ad360bd4f358dbf2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/44523bd8e71e446a87bbf94d5f71a3ac, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/722b721da1f34b5c94461a8e45c1ca27, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/96b16cf73355446dbc7ed40170d238d1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/62280dccd07e4d608a6dd2de70f94404, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/879451fd07d24cff8ad94657ebb6ef43, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e686ff83ce264ed5b51d5f2619b21143, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e5e675b55b974d898ef74ee1f71070d4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/dec89b249c624cf7be25969fb715798d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5ed8e65a805747319ce667f7c57fd9bb, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/3d2630aec2094f24ac805fcc0a9299c2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/87e69ff16391460c9e85f63de48b21f6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/cc14f31466a843259f83bc32da29077c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1040f37feb934a35ba9aee9d6d00226b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1af43ab28441444c9b044817c8d5d447, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c6e4362310e341cb81d07ab4fbcd69cd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d91918dee1b44f87832e38088c40bcde, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/8af12f3f9ff447f59c9c8d3429dfe227] to archive 2023-04-23 05:54:20,499 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:54:20,501 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a49050e3142c4a1c816f48361793bac8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a49050e3142c4a1c816f48361793bac8 2023-04-23 05:54:20,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d96be4f3255c497cbee34c3023ee22e3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d96be4f3255c497cbee34c3023ee22e3 2023-04-23 05:54:20,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ff0e9490797b483aa9231e9c9e558dc6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ff0e9490797b483aa9231e9c9e558dc6 2023-04-23 05:54:20,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/149003e3dc6c4b2993336f2dbeb9d249 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/149003e3dc6c4b2993336f2dbeb9d249 2023-04-23 05:54:20,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5809d075b4f642038cd20d0cff85c082 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5809d075b4f642038cd20d0cff85c082 2023-04-23 05:54:20,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c7439bec5fc04786a7f48781df577aa6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c7439bec5fc04786a7f48781df577aa6 2023-04-23 05:54:20,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/beaaddb75dde46518d79814f03a36d47 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/beaaddb75dde46518d79814f03a36d47 2023-04-23 05:54:20,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/f4db1da8c66347bb8e1b7ee40052a805 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/f4db1da8c66347bb8e1b7ee40052a805 2023-04-23 05:54:20,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a14e4fb37eb44109a342781c3ec49cb4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a14e4fb37eb44109a342781c3ec49cb4 2023-04-23 05:54:20,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/b412b4919e9f4f13b0d1aa27dd0a0f36 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/b412b4919e9f4f13b0d1aa27dd0a0f36 2023-04-23 05:54:20,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ddddc13f5b5a40dfa4b3ce50f6d308c1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/ddddc13f5b5a40dfa4b3ce50f6d308c1 2023-04-23 05:54:20,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a1919cdbc76e4028ab7bd6538902627f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/a1919cdbc76e4028ab7bd6538902627f 2023-04-23 05:54:20,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/aabcf50ab17a4896ad360bd4f358dbf2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/aabcf50ab17a4896ad360bd4f358dbf2 2023-04-23 05:54:20,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/44523bd8e71e446a87bbf94d5f71a3ac to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/44523bd8e71e446a87bbf94d5f71a3ac 2023-04-23 05:54:20,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/722b721da1f34b5c94461a8e45c1ca27 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/722b721da1f34b5c94461a8e45c1ca27 2023-04-23 05:54:20,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/96b16cf73355446dbc7ed40170d238d1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/96b16cf73355446dbc7ed40170d238d1 2023-04-23 05:54:20,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/62280dccd07e4d608a6dd2de70f94404 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/62280dccd07e4d608a6dd2de70f94404 2023-04-23 05:54:20,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/879451fd07d24cff8ad94657ebb6ef43 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/879451fd07d24cff8ad94657ebb6ef43 2023-04-23 05:54:20,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e686ff83ce264ed5b51d5f2619b21143 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e686ff83ce264ed5b51d5f2619b21143 2023-04-23 05:54:20,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e5e675b55b974d898ef74ee1f71070d4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/e5e675b55b974d898ef74ee1f71070d4 2023-04-23 05:54:20,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/dec89b249c624cf7be25969fb715798d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/dec89b249c624cf7be25969fb715798d 2023-04-23 05:54:20,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5ed8e65a805747319ce667f7c57fd9bb to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/5ed8e65a805747319ce667f7c57fd9bb 2023-04-23 05:54:20,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/3d2630aec2094f24ac805fcc0a9299c2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/3d2630aec2094f24ac805fcc0a9299c2 2023-04-23 05:54:20,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/87e69ff16391460c9e85f63de48b21f6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/87e69ff16391460c9e85f63de48b21f6 2023-04-23 05:54:20,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/cc14f31466a843259f83bc32da29077c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/cc14f31466a843259f83bc32da29077c 2023-04-23 05:54:20,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1040f37feb934a35ba9aee9d6d00226b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1040f37feb934a35ba9aee9d6d00226b 2023-04-23 05:54:20,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1af43ab28441444c9b044817c8d5d447 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/1af43ab28441444c9b044817c8d5d447 2023-04-23 05:54:20,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c6e4362310e341cb81d07ab4fbcd69cd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/c6e4362310e341cb81d07ab4fbcd69cd 2023-04-23 05:54:20,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d91918dee1b44f87832e38088c40bcde to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d91918dee1b44f87832e38088c40bcde 2023-04-23 05:54:20,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/8af12f3f9ff447f59c9c8d3429dfe227 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/8af12f3f9ff447f59c9c8d3429dfe227 2023-04-23 05:54:20,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/recovered.edits/580.seqid, newMaxSeqId=580, maxSeqId=1 2023-04-23 05:54:20,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a. 2023-04-23 05:54:20,539 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bfd1e7528fb104543b2ee708b991f02a: 2023-04-23 05:54:20,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:20,541 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=bfd1e7528fb104543b2ee708b991f02a, regionState=CLOSED 2023-04-23 05:54:20,541 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229260541"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229260541"}]},"ts":"1682229260541"} 2023-04-23 05:54:20,544 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-23 05:54:20,544 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure bfd1e7528fb104543b2ee708b991f02a, server=jenkins-hbase4.apache.org,40995,1682229223026 in 1.5770 sec 2023-04-23 05:54:20,546 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-23 05:54:20,546 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bfd1e7528fb104543b2ee708b991f02a, UNASSIGN in 1.5810 sec 2023-04-23 05:54:20,547 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229260546"}]},"ts":"1682229260546"} 2023-04-23 05:54:20,548 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-23 05:54:20,549 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-23 05:54:20,551 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5980 sec 2023-04-23 05:54:21,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-23 05:54:21,065 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-23 05:54:21,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-23 05:54:21,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:21,077 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:21,079 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:21,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-23 05:54:21,084 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:21,087 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/recovered.edits] 2023-04-23 05:54:21,090 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/41d58d6cd5164f3f9278182f74e54c61 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/41d58d6cd5164f3f9278182f74e54c61 2023-04-23 05:54:21,092 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/47d99ec6253f4dbbb335270a796c5f00 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/47d99ec6253f4dbbb335270a796c5f00 2023-04-23 05:54:21,093 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3cd4f8ea6ed4f4f8e26cb99daafdd59 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a3cd4f8ea6ed4f4f8e26cb99daafdd59 2023-04-23 05:54:21,094 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a55bcd916f484c4dbb5fb40e499ec2af to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a55bcd916f484c4dbb5fb40e499ec2af 2023-04-23 05:54:21,096 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a9a79a2d80e4424fb3e32d1ad2baca4c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/A/a9a79a2d80e4424fb3e32d1ad2baca4c 2023-04-23 05:54:21,098 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/470d461500c54c6f9321555de49f582c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/470d461500c54c6f9321555de49f582c 2023-04-23 05:54:21,099 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/579b735d7dd14cdf9ee3694edd66dafc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/579b735d7dd14cdf9ee3694edd66dafc 2023-04-23 05:54:21,101 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/719359dc044c42418dfa53157c977b21 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/719359dc044c42418dfa53157c977b21 2023-04-23 05:54:21,102 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c7167a3f1bb247028ff6c28ef2af87c8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/c7167a3f1bb247028ff6c28ef2af87c8 2023-04-23 05:54:21,103 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/d357098564684606978529421a25d096 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/B/d357098564684606978529421a25d096 2023-04-23 05:54:21,105 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/53e16ee567864a7f979a43881f077d80 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/53e16ee567864a7f979a43881f077d80 2023-04-23 05:54:21,106 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/66ce4efa4c7f46e99e60db3a8a464fe3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/66ce4efa4c7f46e99e60db3a8a464fe3 2023-04-23 05:54:21,107 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/6d398b82795047d3b4a19786c8fe0c06 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/6d398b82795047d3b4a19786c8fe0c06 2023-04-23 05:54:21,109 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/8f8a69f61e914110be87d85a2e35daee to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/8f8a69f61e914110be87d85a2e35daee 2023-04-23 05:54:21,110 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d33504bcd42646d8a9cbe92b709f4280 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/C/d33504bcd42646d8a9cbe92b709f4280 2023-04-23 05:54:21,112 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/recovered.edits/580.seqid to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a/recovered.edits/580.seqid 2023-04-23 05:54:21,113 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/bfd1e7528fb104543b2ee708b991f02a 2023-04-23 05:54:21,113 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:54:21,115 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:21,135 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-23 05:54:21,177 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-23 05:54:21,179 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:21,179 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-23 05:54:21,179 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682229261179"}]},"ts":"9223372036854775807"} 2023-04-23 05:54:21,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-23 05:54:21,182 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-23 05:54:21,182 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bfd1e7528fb104543b2ee708b991f02a, NAME => 'TestAcidGuarantees,,1682229227881.bfd1e7528fb104543b2ee708b991f02a.', STARTKEY => '', ENDKEY => ''}] 2023-04-23 05:54:21,182 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-23 05:54:21,182 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682229261182"}]},"ts":"9223372036854775807"} 2023-04-23 05:54:21,184 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-23 05:54:21,187 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:21,188 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 116 msec 2023-04-23 05:54:21,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-23 05:54:21,382 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-23 05:54:21,399 INFO [Listener at localhost/41845] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=232 (was 208) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x640c037f-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x640c037f-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:40995-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x640c037f-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1954458040_15 at /127.0.0.1:48246 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x640c037f-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=530 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=405 (was 373) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12400 (was 13118) 2023-04-23 05:54:21,411 INFO [Listener at localhost/41845] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=232, OpenFileDescriptor=530, MaxFileDescriptor=60000, SystemLoadAverage=405, ProcessCount=170, AvailableMemoryMB=12400 2023-04-23 05:54:21,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-23 05:54:21,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:54:21,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:21,416 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-23 05:54:21,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-23 05:54:21,417 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-23 05:54:21,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-23 05:54:21,419 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:21,419 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717 empty. 2023-04-23 05:54:21,420 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:21,420 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:54:21,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-23 05:54:21,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-23 05:54:21,829 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-23 05:54:21,831 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ba2bf059880117cc12577e45cc6d7717, NAME => 'TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp 2023-04-23 05:54:22,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-23 05:54:22,238 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:54:22,238 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ba2bf059880117cc12577e45cc6d7717, disabling compactions & flushes 2023-04-23 05:54:22,238 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:22,238 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:22,238 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. after waiting 0 ms 2023-04-23 05:54:22,238 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:22,238 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:22,238 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:22,240 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-23 05:54:22,241 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229262241"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229262241"}]},"ts":"1682229262241"} 2023-04-23 05:54:22,243 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-23 05:54:22,244 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-23 05:54:22,244 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229262244"}]},"ts":"1682229262244"} 2023-04-23 05:54:22,245 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-23 05:54:22,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, ASSIGN}] 2023-04-23 05:54:22,257 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, ASSIGN 2023-04-23 05:54:22,258 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=false 2023-04-23 05:54:22,409 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=ba2bf059880117cc12577e45cc6d7717, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:22,410 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229262409"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229262409"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229262409"}]},"ts":"1682229262409"} 2023-04-23 05:54:22,411 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:54:22,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-23 05:54:22,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:22,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ba2bf059880117cc12577e45cc6d7717, NAME => 'TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:54:22,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:22,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:54:22,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:22,569 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:22,570 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:22,571 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A 2023-04-23 05:54:22,572 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A 2023-04-23 05:54:22,572 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:54:22,572 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ba2bf059880117cc12577e45cc6d7717 columnFamilyName A 2023-04-23 05:54:22,573 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(310): Store=ba2bf059880117cc12577e45cc6d7717/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:54:22,573 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:22,574 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B 2023-04-23 05:54:22,574 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B 2023-04-23 05:54:22,574 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:54:22,575 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ba2bf059880117cc12577e45cc6d7717 columnFamilyName B 2023-04-23 05:54:22,575 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(310): Store=ba2bf059880117cc12577e45cc6d7717/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:54:22,575 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:22,576 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C 2023-04-23 05:54:22,576 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C 2023-04-23 05:54:22,577 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:54:22,577 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ba2bf059880117cc12577e45cc6d7717 columnFamilyName C 2023-04-23 05:54:22,577 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(310): Store=ba2bf059880117cc12577e45cc6d7717/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:54:22,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:22,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:22,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:22,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:54:22,581 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:22,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-23 05:54:22,583 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ba2bf059880117cc12577e45cc6d7717; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67787019, jitterRate=0.010105296969413757}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:54:22,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:22,584 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., pid=18, masterSystemTime=1682229262563 2023-04-23 05:54:22,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:22,586 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:22,586 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=ba2bf059880117cc12577e45cc6d7717, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:22,586 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229262586"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229262586"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229262586"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229262586"}]},"ts":"1682229262586"} 2023-04-23 05:54:22,589 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-23 05:54:22,589 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 in 176 msec 2023-04-23 05:54:22,591 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-23 05:54:22,591 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, ASSIGN in 334 msec 2023-04-23 05:54:22,592 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-23 05:54:22,592 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229262592"}]},"ts":"1682229262592"} 2023-04-23 05:54:22,593 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-23 05:54:22,597 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-23 05:54:22,598 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1840 sec 2023-04-23 05:54:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-23 05:54:23,524 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-23 05:54:23,525 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x3ab850e7 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:23,530 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7053d0b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:23,531 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:23,535 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:23,537 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-23 05:54:23,540 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49506, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-23 05:54:23,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-23 05:54:23,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:54:23,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:23,971 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-23 05:54:23,972 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-23 05:54:23,972 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-23 05:54:23,976 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-23 05:54:23,989 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, REOPEN/MOVE}] 2023-04-23 05:54:23,990 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, REOPEN/MOVE 2023-04-23 05:54:23,990 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ba2bf059880117cc12577e45cc6d7717, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:23,991 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229263990"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229263990"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229263990"}]},"ts":"1682229263990"} 2023-04-23 05:54:23,992 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:54:24,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ba2bf059880117cc12577e45cc6d7717, disabling compactions & flushes 2023-04-23 05:54:24,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. after waiting 0 ms 2023-04-23 05:54:24,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-23 05:54:24,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:24,150 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: ba2bf059880117cc12577e45cc6d7717 to self. 2023-04-23 05:54:24,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,152 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ba2bf059880117cc12577e45cc6d7717, regionState=CLOSED 2023-04-23 05:54:24,152 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229264152"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229264152"}]},"ts":"1682229264152"} 2023-04-23 05:54:24,155 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-23 05:54:24,155 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 in 161 msec 2023-04-23 05:54:24,155 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=true 2023-04-23 05:54:24,306 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ba2bf059880117cc12577e45cc6d7717, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,306 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229264306"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229264306"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229264306"}]},"ts":"1682229264306"} 2023-04-23 05:54:24,308 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:54:24,464 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ba2bf059880117cc12577e45cc6d7717, NAME => 'TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:54:24,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:54:24,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,467 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,468 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A 2023-04-23 05:54:24,468 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A 2023-04-23 05:54:24,468 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:54:24,471 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ba2bf059880117cc12577e45cc6d7717 columnFamilyName A 2023-04-23 05:54:24,473 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(310): Store=ba2bf059880117cc12577e45cc6d7717/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:54:24,473 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,474 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B 2023-04-23 05:54:24,474 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B 2023-04-23 05:54:24,475 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:54:24,475 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ba2bf059880117cc12577e45cc6d7717 columnFamilyName B 2023-04-23 05:54:24,476 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(310): Store=ba2bf059880117cc12577e45cc6d7717/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:54:24,476 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,477 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C 2023-04-23 05:54:24,477 DEBUG [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C 2023-04-23 05:54:24,477 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:54:24,477 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ba2bf059880117cc12577e45cc6d7717 columnFamilyName C 2023-04-23 05:54:24,478 INFO [StoreOpener-ba2bf059880117cc12577e45cc6d7717-1] regionserver.HStore(310): Store=ba2bf059880117cc12577e45cc6d7717/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:54:24,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,479 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:54:24,482 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,483 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ba2bf059880117cc12577e45cc6d7717; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59666220, jitterRate=-0.1109040379524231}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:54:24,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:24,484 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., pid=23, masterSystemTime=1682229264460 2023-04-23 05:54:24,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,486 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ba2bf059880117cc12577e45cc6d7717, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,486 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229264486"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229264486"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229264486"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229264486"}]},"ts":"1682229264486"} 2023-04-23 05:54:24,489 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-23 05:54:24,489 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 in 180 msec 2023-04-23 05:54:24,490 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-23 05:54:24,490 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, REOPEN/MOVE in 500 msec 2023-04-23 05:54:24,493 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-23 05:54:24,493 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 517 msec 2023-04-23 05:54:24,495 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 942 msec 2023-04-23 05:54:24,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-23 05:54:24,504 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x63ebf1e9 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:24,510 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@695af552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:24,511 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x52d98fed to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:24,515 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52f29016, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:24,516 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x18496e17 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:24,520 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f13c14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:24,521 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x25441895 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:24,526 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a5a81b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:24,526 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x40e0a3dc to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:24,530 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45e59be5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:24,530 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x2668d41d to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:24,534 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fbb4986, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:24,535 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x6b416c44 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:24,539 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b69850d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:24,540 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x204b5880 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:24,543 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40883381, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:24,544 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x41173aec to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:24,547 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3be2e516, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:24,553 DEBUG [hconnection-0x44a2174c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:24,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:24,555 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:24,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:24,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:24,557 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:24,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:24,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:24,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:24,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,560 DEBUG [hconnection-0x60be3d52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:24,563 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:24,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:24,564 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,564 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:24,568 DEBUG [hconnection-0x4f1c6982-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:24,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:24,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:24,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:24,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:24,569 DEBUG [hconnection-0x4fedaced-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:24,568 DEBUG [hconnection-0x5246c436-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:24,574 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36222, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:24,574 DEBUG [hconnection-0x61688c4c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:24,574 DEBUG [hconnection-0x331b980a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:24,573 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:24,574 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:24,573 DEBUG [hconnection-0x5c547a65-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:24,573 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:24,576 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:24,577 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:24,579 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36258, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:24,573 DEBUG [hconnection-0x5dcd6e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:24,573 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,577 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36252, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:24,582 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:24,588 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:24,588 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,588 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:24,588 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,589 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:24,589 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:24,591 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36268, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:24,595 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,595 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,596 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:24,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d6847a0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:24,598 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:24,598 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,600 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:24,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:24,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:24,600 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:54:24,600 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,601 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:24,602 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:24,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:24,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:24,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:24,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:24,603 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:24,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:24,608 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:24,608 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:24,608 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:24,608 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:24,608 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:24,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:24,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:24,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:24,610 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,612 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:24,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:24,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:24,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:24,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:24,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:24,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:24,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:24,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,621 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,621 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:24,622 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:24,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:24,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:24,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229324626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229324626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229324630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229324629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229324635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,675 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304230a220c7bb3b943f48de5113b6f0707f7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304230a220c7bb3b943f48de5113b6f0707f7 2023-04-23 05:54:24,704 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/94f410714b734894a875311bcd673f2b 2023-04-23 05:54:24,722 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:24,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:24,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:24,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229324734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229324736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229324737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229324737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229324740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:24,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:24,745 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:24,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:24,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:24,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:24,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,746 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:24,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:24,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:24,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:24,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:24,764 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:24,764 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:24,764 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,764 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:24,764 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:24,764 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,764 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:24,764 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:24,764 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,765 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:24,765 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:24,766 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:24,767 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,767 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:24,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:24,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf23f7a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:24,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:24,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,770 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,770 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:24,770 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:24,770 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:24,772 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:24,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:24,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:24,773 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:24,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:24,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:24,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:24,773 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:24,775 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,775 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,775 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:24,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:24,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:24,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:24,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,776 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,778 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:24,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:24,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:24,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:24,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:24,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:24,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:24,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:24,783 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:24,783 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:24,883 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:24,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:24,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:24,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:24,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:24,890 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:24,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:24,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:24,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:24,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:24,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:24,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:24,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:24,893 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:24,893 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:24,893 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:24,893 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:24,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:24,894 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,894 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:24,894 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:24,894 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:24,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:24,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:24,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:24,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:24,901 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,901 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:24,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:24,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3648c2a6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:24,904 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:24,904 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,905 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,906 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:24,906 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:24,906 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:24,906 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,906 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:24,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:24,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:24,911 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:24,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:24,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:24,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:24,911 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:24,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:24,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:24,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:24,918 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:24,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,920 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:24,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:24,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:24,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:24,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:24,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:24,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,923 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:24,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:24,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,923 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:24,923 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:24,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:24,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:24,924 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:24,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:24,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:24,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229324937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229324938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229324939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229324940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:24,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:24,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229324943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,024 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:25,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:25,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:25,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:25,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:25,028 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:25,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:25,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:25,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,029 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:25,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:25,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:25,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:25,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:25,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:25,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:25,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:25,039 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,039 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,039 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,039 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fd190[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:25,041 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:25,041 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,042 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:25,042 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,042 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:25,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:25,044 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,044 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:25,044 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,044 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:25,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:25,049 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:25,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:25,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:25,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:25,049 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:25,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,053 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:25,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:25,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:25,057 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,057 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:25,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,057 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:25,158 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:25,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:25,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:25,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:25,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:25,164 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:25,164 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/5faf9ba22def4ee2a0a0a918e74c0350 2023-04-23 05:54:25,164 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,164 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:25,164 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:25,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,166 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:25,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:25,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:25,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:25,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:25,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:25,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:25,173 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,173 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:25,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d75d336[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:25,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:25,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,176 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:25,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:25,177 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,177 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:25,179 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:25,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:25,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:25,183 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:25,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:25,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:25,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:25,183 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:25,186 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,186 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,186 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,186 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,188 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,192 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:25,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:25,192 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:25,192 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:25,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229325239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229325243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229325243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229325245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229325249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,292 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:25,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:25,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:25,298 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:25,298 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,298 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:25,298 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:25,299 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,299 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,300 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,301 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,301 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:25,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:25,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:25,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:25,302 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:25,302 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,302 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:25,306 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51200d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:25,308 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:25,308 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,309 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,310 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:25,310 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:25,310 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:25,310 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,310 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:25,311 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,311 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,311 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,311 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:25,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:25,313 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:25,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:25,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:25,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:25,314 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:25,315 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,316 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,323 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,324 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:25,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:25,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,324 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:25,324 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:25,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,425 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:25,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:25,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:25,440 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:25,440 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,440 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:25,440 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:25,441 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,441 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,442 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,443 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,443 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:25,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:25,456 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,456 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:25,456 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:25,456 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:25,456 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:25,458 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,458 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,459 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,460 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:25,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fc9f53[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:25,461 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:25,461 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,464 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,464 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:25,464 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:25,464 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:25,465 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,465 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:25,467 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:25,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:25,470 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:25,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:25,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:25,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:25,470 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:25,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,474 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,476 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,479 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,479 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,480 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,480 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,480 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,480 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,480 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:25,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:25,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:25,480 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,480 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,480 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,480 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:25,480 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:25,580 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:25,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:25,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:25,588 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:25,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:25,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:25,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/d5942f77ea5f4ea1b2deb685e55e045f 2023-04-23 05:54:25,589 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,590 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,594 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:25,595 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,596 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:25,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/94f410714b734894a875311bcd673f2b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/94f410714b734894a875311bcd673f2b 2023-04-23 05:54:25,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:25,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:25,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/94f410714b734894a875311bcd673f2b, entries=150, sequenceid=17, filesize=25.1 K 2023-04-23 05:54:25,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/5faf9ba22def4ee2a0a0a918e74c0350 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5faf9ba22def4ee2a0a0a918e74c0350 2023-04-23 05:54:25,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:25,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:25,608 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,608 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:25,608 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:25,608 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:25,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:25,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:25,612 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:25,613 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,613 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:25,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c83438d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:25,615 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:25,615 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:25,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:25,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:25,624 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5faf9ba22def4ee2a0a0a918e74c0350, entries=150, sequenceid=17, filesize=11.6 K 2023-04-23 05:54:25,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/d5942f77ea5f4ea1b2deb685e55e045f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d5942f77ea5f4ea1b2deb685e55e045f 2023-04-23 05:54:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:25,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:25,639 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:25,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:25,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:25,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:25,640 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:25,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d5942f77ea5f4ea1b2deb685e55e045f, entries=150, sequenceid=17, filesize=11.6 K 2023-04-23 05:54:25,642 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ba2bf059880117cc12577e45cc6d7717 in 1042ms, sequenceid=17, compaction requested=false 2023-04-23 05:54:25,643 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-23 05:54:25,644 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:25,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:25,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:25,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:25,651 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:25,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,651 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:25,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:25,750 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-23 05:54:25,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229264619, max timestamp=1682229265748Num uniques 859; 2023-04-23 05:54:25,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:25,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229264619, max timestamp=1682229265748Num uniques 859; 2023-04-23 05:54:25,751 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:25,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229264619, max timestamp=1682229265748Num uniques 859; 2023-04-23 05:54:25,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:25,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:25,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:25,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:25,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:25,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:25,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:25,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:25,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:25,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:25,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:25,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:25,768 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:25,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:25,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:25,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,770 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,771 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,771 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:25,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682229265759, max timestamp=1682229265781Num uniques 859; 2023-04-23 05:54:25,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-23 05:54:25,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682229265759, max timestamp=1682229265782Num uniques 859; 2023-04-23 05:54:25,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682229265759, max timestamp=1682229265782Num uniques 859; 2023-04-23 05:54:25,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-23 05:54:25,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-23 05:54:25,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:25,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:25,785 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:25,785 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,785 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:25,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:25,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:25,788 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,788 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:25,788 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:25,789 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,789 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:25,789 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:25,789 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,791 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:25,791 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:25,796 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,797 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,797 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,797 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:25,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f6cdd48[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:25,801 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:25,801 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,803 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,803 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:25,803 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:25,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,803 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:25,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:25,805 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:25,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:25,808 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:25,808 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:25,809 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:25,809 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:25,809 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:25,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,811 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,812 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,814 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,815 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:25,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,815 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:25,815 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:25,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229325825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229325829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229325831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229325832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229325833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,916 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:25,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:25,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:25,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:25,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:25,920 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:25,920 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,920 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:25,920 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:25,921 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,921 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,926 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,926 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,926 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,926 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:25,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:25,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:25,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:25,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:25,928 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,928 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:25,928 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:25,928 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:25,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:25,930 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,930 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,932 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,932 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:25,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@634d5e0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:25,933 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:25,933 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,934 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:25,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:25,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229325934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:25,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229325936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229325936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:25,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229325937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229325937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:25,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:25,940 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:25,940 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:25,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:25,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:25,941 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:25,942 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,942 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,942 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:25,943 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:25,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:25,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,950 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:25,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:25,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:25,951 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,951 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:25,951 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:25,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:25,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:25,951 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:25,951 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:25,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:25,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:26,051 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:26,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:26,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:26,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:26,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:26,055 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:26,055 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:26,055 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:26,055 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:26,056 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,056 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,057 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:26,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:26,057 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,057 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:26,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:26,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:26,059 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:26,059 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:26,059 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:26,059 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:26,059 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:26,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:26,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:26,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a6c472[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:26,063 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:26,063 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,064 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,064 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:26,064 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,064 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:26,064 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:26,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:26,067 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:26,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:26,070 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:26,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:26,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:26,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:26,070 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:26,072 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,072 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,072 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:26,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:26,074 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,075 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:26,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:26,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:26,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:26,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:26,078 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:26,079 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,079 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:26,079 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:26,079 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,079 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,079 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,079 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:26,082 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-23 05:54:26,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229326137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229326140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229326141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229326141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229326141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,179 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:26,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:26,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:26,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:26,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:26,183 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:26,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:26,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:26,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:26,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,185 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:26,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:26,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:26,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:26,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:26,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:26,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:26,187 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:26,187 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:26,187 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:26,187 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:26,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:26,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:26,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:26,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:26,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:26,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af7280[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:26,192 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:26,192 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,193 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:26,193 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:26,193 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:26,193 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,193 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:26,196 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:26,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:26,201 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:26,201 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:26,201 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:26,201 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:26,201 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:26,202 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304236b51ed1f98f0487486e316d51e13a82d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236b51ed1f98f0487486e316d51e13a82d 2023-04-23 05:54:26,203 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,203 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:26,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,203 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:26,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,204 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:26,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:26,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:26,209 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=39, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/59b420bb172946c3aff7ff8846259324 2023-04-23 05:54:26,210 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,210 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,210 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,210 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,210 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,210 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:26,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:26,210 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,210 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:26,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:26,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:26,210 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:26,210 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:26,224 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=39 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/ff49df138ec34077a9c496389d1fc6cf 2023-04-23 05:54:26,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=39 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/98c9eb649dad47e98f5db6190f4149f9 2023-04-23 05:54:26,251 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/59b420bb172946c3aff7ff8846259324 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/59b420bb172946c3aff7ff8846259324 2023-04-23 05:54:26,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/59b420bb172946c3aff7ff8846259324, entries=150, sequenceid=39, filesize=25.1 K 2023-04-23 05:54:26,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/ff49df138ec34077a9c496389d1fc6cf as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ff49df138ec34077a9c496389d1fc6cf 2023-04-23 05:54:26,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ff49df138ec34077a9c496389d1fc6cf, entries=150, sequenceid=39, filesize=11.6 K 2023-04-23 05:54:26,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/98c9eb649dad47e98f5db6190f4149f9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/98c9eb649dad47e98f5db6190f4149f9 2023-04-23 05:54:26,269 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/98c9eb649dad47e98f5db6190f4149f9, entries=150, sequenceid=39, filesize=11.6 K 2023-04-23 05:54:26,270 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=161.02 KB/164880 for ba2bf059880117cc12577e45cc6d7717 in 520ms, sequenceid=39, compaction requested=false 2023-04-23 05:54:26,270 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:26,310 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:26,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:26,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:26,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:26,315 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:26,315 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:26,315 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:26,315 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:26,315 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,315 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,317 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:26,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:26,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:26,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:26,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:26,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:26,318 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:26,319 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:26,319 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:26,319 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:26,319 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:26,319 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=161.02 KB heapSize=436.55 KB 2023-04-23 05:54:26,319 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:26,319 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:26,319 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:26,319 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:26,320 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:26,320 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:26,350 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304237a775a9fa1fd42f19a5e609c66940bc6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304237a775a9fa1fd42f19a5e609c66940bc6 2023-04-23 05:54:26,358 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=74, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/194eedbf220e4d72a03b11188a3961ca 2023-04-23 05:54:26,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:26,447 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:26,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229326447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229326448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229326448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229326449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229326449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229326550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229326552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229326552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229326552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229326553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229326754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229326754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229326755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229326756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:26,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229326756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:26,771 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/cc56fedbc3e04b25884a5a4fcaae9cc4 2023-04-23 05:54:27,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229327058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229327060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229327061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229327062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229327064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,188 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/903100e14b83414d9546a4bcda336a55 2023-04-23 05:54:27,194 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/194eedbf220e4d72a03b11188a3961ca as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/194eedbf220e4d72a03b11188a3961ca 2023-04-23 05:54:27,198 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/194eedbf220e4d72a03b11188a3961ca, entries=150, sequenceid=74, filesize=25.1 K 2023-04-23 05:54:27,199 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/cc56fedbc3e04b25884a5a4fcaae9cc4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/cc56fedbc3e04b25884a5a4fcaae9cc4 2023-04-23 05:54:27,209 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/cc56fedbc3e04b25884a5a4fcaae9cc4, entries=150, sequenceid=74, filesize=11.6 K 2023-04-23 05:54:27,210 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/903100e14b83414d9546a4bcda336a55 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/903100e14b83414d9546a4bcda336a55 2023-04-23 05:54:27,218 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/903100e14b83414d9546a4bcda336a55, entries=150, sequenceid=74, filesize=11.6 K 2023-04-23 05:54:27,219 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~436.50 KB/446976, currentSize=33.54 KB/34350 for ba2bf059880117cc12577e45cc6d7717 in 900ms, sequenceid=74, compaction requested=true 2023-04-23 05:54:27,219 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:27,219 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:27,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:27,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:27,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:27,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:27,224 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc079c0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:27,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:27,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:27,227 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:27,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:27,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:27,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:27,229 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:27,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:27,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:27,234 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:27,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:27,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:27,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:27,234 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:27,237 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:27,237 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:27,237 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:27,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:27,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:27,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:27,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:27,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:27,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:27,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:27,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:27,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:27,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:27,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:27,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:27,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:27,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:27,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:27,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:27,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:27,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:27,242 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:27,243 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:27,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:27,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:27,343 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:27,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:27,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:27,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:27,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:27,347 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:27,347 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:27,347 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:27,347 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:27,348 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:27,348 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:27,349 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:27,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:27,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:27,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:27,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:27,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:27,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:27,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:27,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:27,351 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:27,351 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:27,351 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:27,351 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:27,351 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:27,351 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:27,351 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:27,352 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-23 05:54:27,352 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:27,352 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:27,352 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:27,352 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:27,352 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:27,352 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:27,369 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304238e6fd3e0aaf24372b18be78c2a2b55e3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304238e6fd3e0aaf24372b18be78c2a2b55e3 2023-04-23 05:54:27,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:27,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:27,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682229267566, max timestamp=1682229267578Num uniques 859; 2023-04-23 05:54:27,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-23 05:54:27,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682229267566, max timestamp=1682229267578Num uniques 859; 2023-04-23 05:54:27,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-23 05:54:27,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1682229267566, max timestamp=1682229267578Num uniques 859; 2023-04-23 05:54:27,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-23 05:54:27,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229327594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229327595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229327595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229327595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229327596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229327698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229327699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229327700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229327700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229327700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,782 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/6605aee4310f4729bc4647563d61c9b2 2023-04-23 05:54:27,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229327901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229327901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229327901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229327904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:27,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:27,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229327904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,198 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/256e252a84bf4dbd833e3d2d88aa4302 2023-04-23 05:54:28,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229328205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229328206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229328208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229328208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229328210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,612 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/dd5bc9618f874fa195dfea11f34c1c22 2023-04-23 05:54:28,618 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/6605aee4310f4729bc4647563d61c9b2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/6605aee4310f4729bc4647563d61c9b2 2023-04-23 05:54:28,623 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/6605aee4310f4729bc4647563d61c9b2, entries=150, sequenceid=82, filesize=25.1 K 2023-04-23 05:54:28,625 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/256e252a84bf4dbd833e3d2d88aa4302 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/256e252a84bf4dbd833e3d2d88aa4302 2023-04-23 05:54:28,639 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/256e252a84bf4dbd833e3d2d88aa4302, entries=150, sequenceid=82, filesize=11.6 K 2023-04-23 05:54:28,639 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/dd5bc9618f874fa195dfea11f34c1c22 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/dd5bc9618f874fa195dfea11f34c1c22 2023-04-23 05:54:28,644 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/dd5bc9618f874fa195dfea11f34c1c22, entries=150, sequenceid=82, filesize=11.6 K 2023-04-23 05:54:28,645 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for ba2bf059880117cc12577e45cc6d7717 in 1294ms, sequenceid=82, compaction requested=true 2023-04-23 05:54:28,645 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:28,645 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:28,645 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:28,645 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:28,645 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,645 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:28,645 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:28,647 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,647 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:28,647 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,647 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:28,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:28,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d7b3a4d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:28,651 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:28,651 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:28,652 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,652 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:28,653 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:28,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:28,656 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:28,656 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:28,656 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:28,656 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:28,656 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:28,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:28,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:28,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,661 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,663 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,664 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:28,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:28,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:28,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:28,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:28,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:28,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:28,664 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:28,664 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:28,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:28,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:28,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=161.02 KB heapSize=436.55 KB 2023-04-23 05:54:28,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:28,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:28,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:28,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:28,711 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:28,711 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:28,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229328716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229328719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229328718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229328724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229328725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,764 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:28,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:28,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:28,768 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:28,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:28,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:28,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:28,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,770 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:28,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:28,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:28,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:28,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:28,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:28,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:28,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:28,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:28,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:28,772 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:28,772 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:28,773 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:28,773 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:28,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:28,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:28,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:28,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:28,774 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,774 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:28,774 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,775 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:28,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69dae2c4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:28,776 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:28,776 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,778 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:28,779 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,779 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:28,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,780 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:28,780 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:28,780 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:28,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:28,782 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:28,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:28,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:28,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:28,782 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:28,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:28,784 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,786 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:28,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,788 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,788 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,788 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:28,788 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:28,788 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:28,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:28,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:28,788 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:28,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:28,788 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,788 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,789 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,789 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:28,789 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:28,789 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:28,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:28,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229328826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229328826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229328826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229328826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:28,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229328829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,889 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:28,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:28,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:28,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:28,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:28,892 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:28,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:28,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:28,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:28,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:28,896 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:28,896 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,896 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:28,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:28,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:28,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:28,897 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:28,897 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:28,897 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:28,897 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:28,898 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:28,898 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:28,899 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:28,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:28,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:28,901 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:28,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@161ea60d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:28,901 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,902 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:28,903 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,903 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:28,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:28,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:28,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:28,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:28,912 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:28,912 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:28,912 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:28,912 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:28,912 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:28,913 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,913 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:28,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,913 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:28,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,915 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:28,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:28,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:28,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:28,918 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:28,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:28,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:28,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:28,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:28,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:28,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:28,918 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:28,918 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:29,018 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:29,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:29,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:29,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:29,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:29,022 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:29,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:29,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:29,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,024 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,024 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,025 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:29,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:29,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:29,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:29,026 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,026 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:29,026 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:29,026 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:29,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229329028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,029 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229329028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229329028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229329029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:29,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@787ec377[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:29,031 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:29,031 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,034 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:29,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:29,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:29,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:29,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229329033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:29,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:29,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:29,038 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:29,038 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:29,038 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:29,038 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:29,038 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:29,043 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,043 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,044 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,044 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,049 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,049 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,049 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:29,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:29,049 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,049 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,050 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:29,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,050 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:29,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,132 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042319a897453e5b42c28caa669481491880 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042319a897453e5b42c28caa669481491880 2023-04-23 05:54:29,140 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/d3b2d635e945402b970e781446366a1c 2023-04-23 05:54:29,150 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:29,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:29,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:29,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:29,155 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:29,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:29,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:29,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,158 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,158 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:29,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:29,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:29,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:29,168 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,168 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:29,168 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:29,168 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:29,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:29,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:29,173 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,173 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:29,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c432767[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:29,176 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:29,176 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,177 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,177 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:29,177 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:29,177 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:29,177 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,178 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:29,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:29,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:29,184 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:29,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:29,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:29,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:29,184 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:29,186 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,186 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,186 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,188 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,191 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:29,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:29,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,191 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:29,191 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:29,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,291 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:29,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:29,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:29,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:29,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:29,295 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:29,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:29,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:29,296 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,296 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,297 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:29,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:29,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:29,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:29,298 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:29,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:29,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:29,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:29,299 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,299 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:29,299 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:29,299 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:29,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:29,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:29,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:29,303 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,304 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,304 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:29,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@523124e9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:29,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:29,306 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,307 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:29,307 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,307 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:29,311 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:29,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:29,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:29,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:29,311 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:29,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,319 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,323 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,323 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:29,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,323 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,323 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,323 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,323 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,324 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:29,323 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,324 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:29,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,324 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229329330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229329332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229329333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229329334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229329336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,424 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:29,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:29,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:29,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:29,427 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:29,427 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,427 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:29,427 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:29,428 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,428 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,429 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,429 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,429 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,430 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:29,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:29,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:29,431 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,431 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:29,431 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:29,431 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:29,432 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:29,433 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317cde6c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:29,435 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:29,435 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,437 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:29,437 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,437 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:29,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:29,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:29,438 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:29,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:29,441 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:29,441 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:29,441 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:29,441 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:29,441 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:29,442 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,442 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,443 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,443 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,444 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,446 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,446 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,447 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:29,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:29,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,447 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:29,447 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:29,547 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:29,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:29,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:29,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:29,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:29,551 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:29,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:29,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:29,552 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,552 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,554 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,554 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,555 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,555 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:29,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:29,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:29,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:29,556 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,556 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:29,556 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:29,557 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:29,557 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:29,558 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,558 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,559 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:29,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f17d64f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:29,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:29,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,561 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:29,562 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:29,562 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:29,562 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,562 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:29,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:29,566 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:29,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:29,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:29,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:29,566 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:29,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,570 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:29,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,572 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:29,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:29,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,573 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:29,573 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:29,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/2cd9ebb233534373837d12b03452f736 2023-04-23 05:54:29,673 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:29,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:29,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:29,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:29,677 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:29,677 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,677 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:29,677 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:29,677 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,679 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:29,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:29,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:29,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:29,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:29,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:29,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:29,680 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,680 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:29,681 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:29,681 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:29,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:29,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:29,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:29,682 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:29,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a1aa6d5[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:29,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:29,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:29,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:29,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:29,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:29,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:29,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,688 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,688 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:29,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:29,695 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:29,695 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:29,695 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:29,695 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:29,695 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:29,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,697 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:29,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:29,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,702 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,702 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:29,702 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:29,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,803 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:29,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:29,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:29,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:29,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:29,806 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:29,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:29,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:29,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,808 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,809 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,809 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:29,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:29,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:29,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:29,810 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,810 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:29,810 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:29,810 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:29,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:29,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,812 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:29,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1157c513[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:29,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:29,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,815 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:29,815 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:29,815 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:29,815 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,815 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:29,818 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:29,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:29,821 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:29,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:29,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:29,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:29,821 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:29,822 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,822 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,822 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,825 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,827 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:29,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:29,828 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,828 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:29,828 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:29,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229329834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229329837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229329838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229329840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:29,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229329843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,928 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:29,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:29,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:29,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:29,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:29,933 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:29,933 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,933 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:29,933 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:29,934 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,934 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,935 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:29,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:29,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:29,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:29,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:29,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:29,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:29,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:29,937 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,937 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:29,937 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:29,937 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:29,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:29,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:29,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:29,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:29,940 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,940 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,940 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,940 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:29,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@468fe6eb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:29,942 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:29,942 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,943 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,943 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:29,943 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:29,943 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:29,944 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,944 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:29,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:29,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:29,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:29,947 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:29,947 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:29,947 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:29,947 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:29,947 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:29,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:29,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,949 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,949 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:29,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,950 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:29,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:29,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:29,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:29,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:29,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:29,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:29,954 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:29,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:29,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:29,954 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:29,954 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:29,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,955 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:29,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:29,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:29,955 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:29,986 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/f8f3fc92e1384ed6add0f5ac9ffc9654 2023-04-23 05:54:29,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/d3b2d635e945402b970e781446366a1c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d3b2d635e945402b970e781446366a1c 2023-04-23 05:54:29,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d3b2d635e945402b970e781446366a1c, entries=200, sequenceid=117, filesize=31.9 K 2023-04-23 05:54:29,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/2cd9ebb233534373837d12b03452f736 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/2cd9ebb233534373837d12b03452f736 2023-04-23 05:54:30,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/2cd9ebb233534373837d12b03452f736, entries=150, sequenceid=117, filesize=11.6 K 2023-04-23 05:54:30,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/f8f3fc92e1384ed6add0f5ac9ffc9654 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f8f3fc92e1384ed6add0f5ac9ffc9654 2023-04-23 05:54:30,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f8f3fc92e1384ed6add0f5ac9ffc9654, entries=150, sequenceid=117, filesize=11.6 K 2023-04-23 05:54:30,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~436.50 KB/446976, currentSize=33.54 KB/34350 for ba2bf059880117cc12577e45cc6d7717 in 1310ms, sequenceid=117, compaction requested=true 2023-04-23 05:54:30,021 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:30,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:30,021 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:30,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:30,021 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:30,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:30,024 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:30,024 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 135670 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:30,024 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/B is initiating minor compaction (all files) 2023-04-23 05:54:30,024 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/A is initiating minor compaction (all files) 2023-04-23 05:54:30,024 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/B in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:30,024 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/A in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:30,024 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5faf9ba22def4ee2a0a0a918e74c0350, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ff49df138ec34077a9c496389d1fc6cf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/cc56fedbc3e04b25884a5a4fcaae9cc4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/256e252a84bf4dbd833e3d2d88aa4302, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/2cd9ebb233534373837d12b03452f736] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=58.0 K 2023-04-23 05:54:30,024 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/94f410714b734894a875311bcd673f2b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/59b420bb172946c3aff7ff8846259324, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/194eedbf220e4d72a03b11188a3961ca, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/6605aee4310f4729bc4647563d61c9b2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d3b2d635e945402b970e781446366a1c] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=132.5 K 2023-04-23 05:54:30,025 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 5faf9ba22def4ee2a0a0a918e74c0350, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682229264596 2023-04-23 05:54:30,025 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 94f410714b734894a875311bcd673f2b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682229264596 2023-04-23 05:54:30,025 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ff49df138ec34077a9c496389d1fc6cf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=39, earliestPutTs=1682229264619 2023-04-23 05:54:30,025 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 59b420bb172946c3aff7ff8846259324, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=39, earliestPutTs=1682229264619 2023-04-23 05:54:30,026 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting cc56fedbc3e04b25884a5a4fcaae9cc4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1682229265828 2023-04-23 05:54:30,026 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 194eedbf220e4d72a03b11188a3961ca, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1682229265828 2023-04-23 05:54:30,026 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 256e252a84bf4dbd833e3d2d88aa4302, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682229266445 2023-04-23 05:54:30,026 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 6605aee4310f4729bc4647563d61c9b2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682229266445 2023-04-23 05:54:30,026 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2cd9ebb233534373837d12b03452f736, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682229267593 2023-04-23 05:54:30,027 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting d3b2d635e945402b970e781446366a1c, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682229267593 2023-04-23 05:54:30,037 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#B#compaction#120 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:30,043 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#A#compaction#121 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:30,055 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:30,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:30,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:30,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:30,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:30,065 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:30,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:30,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:30,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:30,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:30,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:30,067 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:30,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:30,067 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:30,067 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,067 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:30,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:30,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:30,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:30,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:30,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:30,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:30,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:30,072 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:30,076 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/e979bea0be0047bcaf8aa60b00a1fcde as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/e979bea0be0047bcaf8aa60b00a1fcde 2023-04-23 05:54:30,080 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:30,080 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:30,080 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:30,080 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:30,080 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-23 05:54:30,080 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:30,080 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:30,081 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:30,081 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:30,081 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:30,081 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:30,084 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/B of ba2bf059880117cc12577e45cc6d7717 into e979bea0be0047bcaf8aa60b00a1fcde(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:30,084 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:30,084 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/B, priority=11, startTime=1682229270021; duration=0sec 2023-04-23 05:54:30,084 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:30,084 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:30,086 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:30,087 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/C is initiating minor compaction (all files) 2023-04-23 05:54:30,087 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/C in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:30,087 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d5942f77ea5f4ea1b2deb685e55e045f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/98c9eb649dad47e98f5db6190f4149f9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/903100e14b83414d9546a4bcda336a55, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/dd5bc9618f874fa195dfea11f34c1c22, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f8f3fc92e1384ed6add0f5ac9ffc9654] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=58.0 K 2023-04-23 05:54:30,087 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d5942f77ea5f4ea1b2deb685e55e045f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682229264596 2023-04-23 05:54:30,087 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 98c9eb649dad47e98f5db6190f4149f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=39, earliestPutTs=1682229264619 2023-04-23 05:54:30,088 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 903100e14b83414d9546a4bcda336a55, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1682229265828 2023-04-23 05:54:30,088 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting dd5bc9618f874fa195dfea11f34c1c22, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682229266445 2023-04-23 05:54:30,089 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f8f3fc92e1384ed6add0f5ac9ffc9654, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682229267593 2023-04-23 05:54:30,111 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#C#compaction#123 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:30,131 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423a1e4b4df4e7f4b2caf7585f8236467e3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a1e4b4df4e7f4b2caf7585f8236467e3 2023-04-23 05:54:30,145 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=125, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/dc1bd79878d4466eb2c9b1edc59580ff 2023-04-23 05:54:30,501 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/ffc58ff0646e4a0da5a522000417fd63 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/ffc58ff0646e4a0da5a522000417fd63 2023-04-23 05:54:30,506 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/A of ba2bf059880117cc12577e45cc6d7717 into ffc58ff0646e4a0da5a522000417fd63(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:30,506 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:30,506 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/A, priority=11, startTime=1682229270021; duration=0sec 2023-04-23 05:54:30,506 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:30,526 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/745d70a5ee02400abde37447c50d2440 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/745d70a5ee02400abde37447c50d2440 2023-04-23 05:54:30,531 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/C of ba2bf059880117cc12577e45cc6d7717 into 745d70a5ee02400abde37447c50d2440(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:30,531 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:30,531 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/C, priority=11, startTime=1682229270021; duration=0sec 2023-04-23 05:54:30,531 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:30,563 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/d1a176a37cac4e84b9641aad65735004 2023-04-23 05:54:30,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:30,839 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:30,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229270855, max timestamp=1682229270859Num uniques 859; 2023-04-23 05:54:30,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:30,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229270855, max timestamp=1682229270859Num uniques 859; 2023-04-23 05:54:30,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:30,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229270855, max timestamp=1682229270859Num uniques 859; 2023-04-23 05:54:30,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:30,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229330874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229330875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229330875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229330876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229330877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,986 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/9fd17d15e46f470ea8183cadc59130c1 2023-04-23 05:54:30,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229330986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229330986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229330986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229330986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:30,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229330986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:30,992 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/dc1bd79878d4466eb2c9b1edc59580ff as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc1bd79878d4466eb2c9b1edc59580ff 2023-04-23 05:54:30,997 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc1bd79878d4466eb2c9b1edc59580ff, entries=150, sequenceid=125, filesize=25.1 K 2023-04-23 05:54:30,998 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/d1a176a37cac4e84b9641aad65735004 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d1a176a37cac4e84b9641aad65735004 2023-04-23 05:54:31,003 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d1a176a37cac4e84b9641aad65735004, entries=150, sequenceid=125, filesize=11.6 K 2023-04-23 05:54:31,004 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/9fd17d15e46f470ea8183cadc59130c1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/9fd17d15e46f470ea8183cadc59130c1 2023-04-23 05:54:31,008 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/9fd17d15e46f470ea8183cadc59130c1, entries=150, sequenceid=125, filesize=11.6 K 2023-04-23 05:54:31,009 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for ba2bf059880117cc12577e45cc6d7717 in 929ms, sequenceid=125, compaction requested=false 2023-04-23 05:54:31,009 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:31,009 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:31,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:31,011 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:31,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:31,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:31,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@472bf5db[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:31,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:31,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:31,014 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:31,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:31,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:31,015 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:31,014 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:31,015 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:31,017 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:31,021 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:31,021 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:31,021 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:31,021 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:31,021 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:31,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:31,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:31,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:31,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:31,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:31,024 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:31,025 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,030 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:31,030 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:31,030 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:31,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:31,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:31,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:31,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:31,030 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:31,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:31,031 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:31,031 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:31,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:31,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:31,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:31,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:31,131 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:31,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:31,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:31,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:31,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:31,135 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:31,135 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:31,135 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:31,135 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:31,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:31,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:31,138 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:31,138 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:31,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:31,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:31,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:31,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:31,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:31,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:31,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:31,140 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:31,140 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:31,140 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:31,140 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:31,140 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:31,140 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:31,140 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:31,140 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-23 05:54:31,141 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:31,141 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:31,141 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:31,141 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:31,141 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:31,141 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:31,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:31,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:31,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229331194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229331194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229331195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229331196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229331197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229331298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229331298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229331299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229331299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229331302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229331501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229331501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229331502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229331502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229331506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,559 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423e01d8c94343745a6a378b9fd109e4100 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e01d8c94343745a6a378b9fd109e4100 2023-04-23 05:54:31,570 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/05b11430917d46bb914a549e774bb5e9 2023-04-23 05:54:31,597 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/ec26691504ad4eaf9ea81348fa7398c2 2023-04-23 05:54:31,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229331806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229331807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229331808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229331809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:31,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:31,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229331810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,018 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/bc8cb2d546794b8188aabf949d7ea9bb 2023-04-23 05:54:32,024 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/05b11430917d46bb914a549e774bb5e9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/05b11430917d46bb914a549e774bb5e9 2023-04-23 05:54:32,029 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/05b11430917d46bb914a549e774bb5e9, entries=150, sequenceid=165, filesize=25.3 K 2023-04-23 05:54:32,030 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/ec26691504ad4eaf9ea81348fa7398c2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ec26691504ad4eaf9ea81348fa7398c2 2023-04-23 05:54:32,036 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ec26691504ad4eaf9ea81348fa7398c2, entries=150, sequenceid=165, filesize=11.8 K 2023-04-23 05:54:32,037 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/bc8cb2d546794b8188aabf949d7ea9bb as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/bc8cb2d546794b8188aabf949d7ea9bb 2023-04-23 05:54:32,041 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/bc8cb2d546794b8188aabf949d7ea9bb, entries=150, sequenceid=165, filesize=11.8 K 2023-04-23 05:54:32,045 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for ba2bf059880117cc12577e45cc6d7717 in 905ms, sequenceid=165, compaction requested=true 2023-04-23 05:54:32,045 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:32,045 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:32,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:32,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:32,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:32,046 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:32,047 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,047 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:32,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:32,048 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,048 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:32,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:32,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:32,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5818cfe5[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:32,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:32,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:32,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:32,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:32,052 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,052 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:32,052 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:32,052 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:32,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:32,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:32,054 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:32,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:32,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:32,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:32,054 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:32,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:32,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,057 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:32,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:32,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:32,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,059 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:32,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:32,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:32,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:32,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:32,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:32,061 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:32,061 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:32,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:32,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:32,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:32,061 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:32,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,161 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:32,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:32,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:32,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:32,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:32,165 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:32,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:32,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:32,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:32,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:32,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:32,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:32,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:32,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:32,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:32,169 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:32,169 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:32,169 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-23 05:54:32,169 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:32,169 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:32,170 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:32,170 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:32,170 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:32,170 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:32,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:32,313 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:32,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229272321, max timestamp=1682229272326Num uniques 859; 2023-04-23 05:54:32,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229272321, max timestamp=1682229272326Num uniques 859; 2023-04-23 05:54:32,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:32,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:32,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229272321, max timestamp=1682229272326Num uniques 859; 2023-04-23 05:54:32,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:32,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229332343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229332344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229332345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229332346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229332346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229332448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229332448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229332448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229332450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229332451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,587 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304236562799b1bc447f9abe067ac2fde2bbd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236562799b1bc447f9abe067ac2fde2bbd 2023-04-23 05:54:32,601 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=173, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/c9ef2589658b4aefb2c230a18abf0435 2023-04-23 05:54:32,619 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/4858d24e32f74581812717d207dfbe66 2023-04-23 05:54:32,638 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/04a87b1145ee42dbbf25cd1d70b8e91b 2023-04-23 05:54:32,645 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/c9ef2589658b4aefb2c230a18abf0435 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c9ef2589658b4aefb2c230a18abf0435 2023-04-23 05:54:32,654 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c9ef2589658b4aefb2c230a18abf0435, entries=150, sequenceid=173, filesize=25.3 K 2023-04-23 05:54:32,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229332653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229332655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,656 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/4858d24e32f74581812717d207dfbe66 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/4858d24e32f74581812717d207dfbe66 2023-04-23 05:54:32,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229332655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229332655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,661 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/4858d24e32f74581812717d207dfbe66, entries=150, sequenceid=173, filesize=11.8 K 2023-04-23 05:54:32,662 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/04a87b1145ee42dbbf25cd1d70b8e91b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/04a87b1145ee42dbbf25cd1d70b8e91b 2023-04-23 05:54:32,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229332657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,668 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/04a87b1145ee42dbbf25cd1d70b8e91b, entries=150, sequenceid=173, filesize=11.8 K 2023-04-23 05:54:32,669 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for ba2bf059880117cc12577e45cc6d7717 in 500ms, sequenceid=173, compaction requested=true 2023-04-23 05:54:32,669 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:32,669 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:32,671 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:32,672 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,672 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:32,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7748f6a8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:32,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:32,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:32,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:32,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:32,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:32,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:32,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:32,679 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:32,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:32,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:32,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:32,679 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:32,680 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:32,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:32,682 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:32,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:32,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:32,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:32,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:32,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:32,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:32,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:32,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:32,686 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:32,686 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:32,786 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:32,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:32,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:32,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:32,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:32,790 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:32,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:32,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:32,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:32,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,791 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,792 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:32,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:32,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:32,793 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,793 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:32,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:32,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:32,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:32,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:32,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:32,794 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:32,794 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:32,794 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:32,794 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:32,794 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:32,795 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:32,795 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:32,795 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-23 05:54:32,795 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:32,795 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:32,795 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:32,795 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:32,795 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:32,795 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:32,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:32,960 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:32,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229332962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229332962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229332962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229332963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:32,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:32,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229332966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229333064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229333064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229333065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229333066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,218 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042347b891a6ab7a4e8f962f6e9074c93361 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042347b891a6ab7a4e8f962f6e9074c93361 2023-04-23 05:54:33,224 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=210, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/75d7e5a5629c48f99b68e0a500aad549 2023-04-23 05:54:33,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229333267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229333268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229333268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229333270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229333471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229333571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229333572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229333572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:33,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229333576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:33,636 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/7924ae9cf2d74cf6b41444ea888f274b 2023-04-23 05:54:34,049 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/894557bf164b456f931225038d21cc34 2023-04-23 05:54:34,055 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/75d7e5a5629c48f99b68e0a500aad549 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/75d7e5a5629c48f99b68e0a500aad549 2023-04-23 05:54:34,061 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/75d7e5a5629c48f99b68e0a500aad549, entries=150, sequenceid=210, filesize=25.3 K 2023-04-23 05:54:34,062 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/7924ae9cf2d74cf6b41444ea888f274b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7924ae9cf2d74cf6b41444ea888f274b 2023-04-23 05:54:34,067 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7924ae9cf2d74cf6b41444ea888f274b, entries=150, sequenceid=210, filesize=11.8 K 2023-04-23 05:54:34,068 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/894557bf164b456f931225038d21cc34 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/894557bf164b456f931225038d21cc34 2023-04-23 05:54:34,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229334073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,078 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/894557bf164b456f931225038d21cc34, entries=150, sequenceid=210, filesize=11.8 K 2023-04-23 05:54:34,079 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for ba2bf059880117cc12577e45cc6d7717 in 1284ms, sequenceid=210, compaction requested=true 2023-04-23 05:54:34,079 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:34,079 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,079 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:34,079 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:34,079 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,080 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:34,080 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:34,084 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:34,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e49ee14[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:34,089 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:34,089 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,091 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:34,091 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:34,091 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:34,091 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,091 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:34,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:34,092 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:54:34,092 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:34,093 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:34,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:34,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:34,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:34,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:34,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:34,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:34,099 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:34,099 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:34,099 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:34,099 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:34,099 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:34,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:34,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,110 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:34,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:34,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:34,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:34,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:34,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:34,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:34,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:34,114 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:34,114 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:34,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:34,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229274130, max timestamp=1682229274144Num uniques 859; 2023-04-23 05:54:34,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:34,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229274130, max timestamp=1682229274144Num uniques 859; 2023-04-23 05:54:34,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:34,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229274130, max timestamp=1682229274144Num uniques 859; 2023-04-23 05:54:34,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:34,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229334171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229334172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229334173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,214 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:34,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:34,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:34,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:34,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:34,217 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:34,217 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:34,217 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:34,217 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:34,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,219 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:34,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:34,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:34,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:34,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:34,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:34,221 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,221 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:34,221 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:34,221 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,221 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:34,221 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:34,221 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,221 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:34,221 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:34,227 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:34,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56544f2b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:34,229 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:34,229 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:34,230 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:34,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:34,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:34,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:34,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:34,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:34,234 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:34,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:34,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:34,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:34,234 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:34,235 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,236 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,236 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:34,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,236 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:34,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:34,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:34,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:34,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:34,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:34,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:34,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:34,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,241 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:34,241 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:34,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229334274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229334274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229334276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,341 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:34,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:34,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:34,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:34,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:34,345 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:34,345 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:34,345 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:34,345 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:34,346 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,346 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,347 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:34,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:34,347 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,347 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:34,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:34,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:34,348 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:34,349 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,349 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:34,349 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:34,349 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:34,349 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:34,350 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,350 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,350 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,350 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:34,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@482dd5e3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:34,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:34,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,353 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,353 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:34,353 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:34,353 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:34,353 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,353 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:34,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:34,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:34,357 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:34,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:34,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:34,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:34,358 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:34,359 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,359 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,359 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,359 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,361 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:34,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:34,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:34,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:34,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:34,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:34,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,363 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:34,363 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:34,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:34,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:34,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,464 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:34,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:34,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:34,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:34,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:34,467 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:34,467 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:34,467 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:34,467 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:34,468 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,468 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,469 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:34,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:34,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:34,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:34,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:34,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:34,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:34,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:34,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:34,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:34,471 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,471 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:34,471 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:34,471 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:34,473 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:34,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786087f8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:34,475 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:34,475 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,476 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,476 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:34,476 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:34,476 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:34,477 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,477 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:34,477 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:34,477 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:34,478 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:34,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229334476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,478 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229334477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229334478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229334478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:34,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:34,480 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:34,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:34,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:34,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:34,481 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:34,482 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,482 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:34,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,482 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:34,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:34,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,484 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,487 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,487 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,487 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:34,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:34,487 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:34,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:34,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:34,487 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:34,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:34,488 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:34,488 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,488 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:34,488 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,488 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:34,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:34,488 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:34,511 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304238e0b5d0fe9604a75a0a84e55a77dccb1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304238e0b5d0fe9604a75a0a84e55a77dccb1 2023-04-23 05:54:34,519 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/7b78ceb645f144ff8bdcb06d4029482b 2023-04-23 05:54:34,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/39c7a28c49604af2b974ca8026118c46 2023-04-23 05:54:34,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/03b6997d60b14ad58a85a3f5a257c9b9 2023-04-23 05:54:34,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/7b78ceb645f144ff8bdcb06d4029482b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/7b78ceb645f144ff8bdcb06d4029482b 2023-04-23 05:54:34,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/7b78ceb645f144ff8bdcb06d4029482b, entries=250, sequenceid=221, filesize=38.9 K 2023-04-23 05:54:34,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/39c7a28c49604af2b974ca8026118c46 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/39c7a28c49604af2b974ca8026118c46 2023-04-23 05:54:34,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/39c7a28c49604af2b974ca8026118c46, entries=150, sequenceid=221, filesize=11.8 K 2023-04-23 05:54:34,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/03b6997d60b14ad58a85a3f5a257c9b9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/03b6997d60b14ad58a85a3f5a257c9b9 2023-04-23 05:54:34,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/03b6997d60b14ad58a85a3f5a257c9b9, entries=150, sequenceid=221, filesize=11.8 K 2023-04-23 05:54:34,584 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ba2bf059880117cc12577e45cc6d7717 in 492ms, sequenceid=221, compaction requested=true 2023-04-23 05:54:34,585 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:34,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:34,585 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:54:34,585 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:54:34,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:34,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:34,587 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 169232 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:54:34,587 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/A is initiating minor compaction (all files) 2023-04-23 05:54:34,587 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/A in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,587 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72088 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:54:34,587 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/ffc58ff0646e4a0da5a522000417fd63, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc1bd79878d4466eb2c9b1edc59580ff, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/05b11430917d46bb914a549e774bb5e9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c9ef2589658b4aefb2c230a18abf0435, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/75d7e5a5629c48f99b68e0a500aad549, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/7b78ceb645f144ff8bdcb06d4029482b] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=165.3 K 2023-04-23 05:54:34,587 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/B is initiating minor compaction (all files) 2023-04-23 05:54:34,587 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/B in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,587 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/e979bea0be0047bcaf8aa60b00a1fcde, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d1a176a37cac4e84b9641aad65735004, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ec26691504ad4eaf9ea81348fa7398c2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/4858d24e32f74581812717d207dfbe66, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7924ae9cf2d74cf6b41444ea888f274b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/39c7a28c49604af2b974ca8026118c46] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=70.4 K 2023-04-23 05:54:34,588 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting ffc58ff0646e4a0da5a522000417fd63, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682229267593 2023-04-23 05:54:34,588 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e979bea0be0047bcaf8aa60b00a1fcde, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682229267593 2023-04-23 05:54:34,588 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:34,588 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting dc1bd79878d4466eb2c9b1edc59580ff, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1682229268713 2023-04-23 05:54:34,589 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 05b11430917d46bb914a549e774bb5e9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682229270870 2023-04-23 05:54:34,589 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d1a176a37cac4e84b9641aad65735004, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1682229268713 2023-04-23 05:54:34,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:34,589 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c9ef2589658b4aefb2c230a18abf0435, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682229271194 2023-04-23 05:54:34,589 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ec26691504ad4eaf9ea81348fa7398c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682229270870 2023-04-23 05:54:34,590 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 75d7e5a5629c48f99b68e0a500aad549, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682229272343 2023-04-23 05:54:34,590 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 4858d24e32f74581812717d207dfbe66, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682229271194 2023-04-23 05:54:34,591 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 7b78ceb645f144ff8bdcb06d4029482b, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682229272959 2023-04-23 05:54:34,591 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7924ae9cf2d74cf6b41444ea888f274b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682229272343 2023-04-23 05:54:34,591 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 39c7a28c49604af2b974ca8026118c46, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682229272959 2023-04-23 05:54:34,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:34,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:34,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:34,619 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:34,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:34,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:34,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:34,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,622 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#B#compaction#139 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:34,622 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:34,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:34,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:34,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:34,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:34,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:34,627 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#A#compaction#138 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:34,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:34,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:34,628 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,628 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:34,628 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-23 05:54:34,628 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:34,628 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:34,629 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:34,629 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:34,629 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:34,629 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:34,651 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/ed82e7b138314f7e868154868f720ad4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ed82e7b138314f7e868154868f720ad4 2023-04-23 05:54:34,659 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/B of ba2bf059880117cc12577e45cc6d7717 into ed82e7b138314f7e868154868f720ad4(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:34,659 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:34,659 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/B, priority=10, startTime=1682229274585; duration=0sec 2023-04-23 05:54:34,659 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:34,659 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:54:34,664 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72088 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:54:34,664 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/C is initiating minor compaction (all files) 2023-04-23 05:54:34,671 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/C in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:34,671 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/745d70a5ee02400abde37447c50d2440, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/9fd17d15e46f470ea8183cadc59130c1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/bc8cb2d546794b8188aabf949d7ea9bb, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/04a87b1145ee42dbbf25cd1d70b8e91b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/894557bf164b456f931225038d21cc34, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/03b6997d60b14ad58a85a3f5a257c9b9] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=70.4 K 2023-04-23 05:54:34,672 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 745d70a5ee02400abde37447c50d2440, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682229267593 2023-04-23 05:54:34,678 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9fd17d15e46f470ea8183cadc59130c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1682229268713 2023-04-23 05:54:34,679 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting bc8cb2d546794b8188aabf949d7ea9bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682229270870 2023-04-23 05:54:34,679 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304233af71a45465046bd980d6ec7a7cdb295 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233af71a45465046bd980d6ec7a7cdb295 2023-04-23 05:54:34,680 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 04a87b1145ee42dbbf25cd1d70b8e91b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1682229271194 2023-04-23 05:54:34,680 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 894557bf164b456f931225038d21cc34, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682229272343 2023-04-23 05:54:34,683 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 03b6997d60b14ad58a85a3f5a257c9b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682229272959 2023-04-23 05:54:34,702 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/dc7083c72ed648aa8403541f4d9fb1e6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc7083c72ed648aa8403541f4d9fb1e6 2023-04-23 05:54:34,703 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=255, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/d95d2f7676194bbda2ee2b43cef99282 2023-04-23 05:54:34,727 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#C#compaction#141 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:34,729 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/A of ba2bf059880117cc12577e45cc6d7717 into dc7083c72ed648aa8403541f4d9fb1e6(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:34,729 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:34,730 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/A, priority=10, startTime=1682229274585; duration=0sec 2023-04-23 05:54:34,730 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:34,757 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/3525f395ba3b48d5a565b49df909c29a 2023-04-23 05:54:34,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:34,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:34,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36268 deadline: 1682229334787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229334788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229334790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229334892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36260 deadline: 1682229334892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:34,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:34,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229334896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229335080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229335095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229335096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1682229335101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,160 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/acdbb257c04f4d21a063be3bc0262cba as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/acdbb257c04f4d21a063be3bc0262cba 2023-04-23 05:54:35,166 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/C of ba2bf059880117cc12577e45cc6d7717 into acdbb257c04f4d21a063be3bc0262cba(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:35,166 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:35,166 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/C, priority=10, startTime=1682229274585; duration=0sec 2023-04-23 05:54:35,166 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:35,172 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/a6ec29aa2c314a79838e2f72474ec403 2023-04-23 05:54:35,180 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/d95d2f7676194bbda2ee2b43cef99282 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d95d2f7676194bbda2ee2b43cef99282 2023-04-23 05:54:35,187 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d95d2f7676194bbda2ee2b43cef99282, entries=150, sequenceid=255, filesize=25.3 K 2023-04-23 05:54:35,188 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/3525f395ba3b48d5a565b49df909c29a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3525f395ba3b48d5a565b49df909c29a 2023-04-23 05:54:35,201 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3525f395ba3b48d5a565b49df909c29a, entries=150, sequenceid=255, filesize=11.8 K 2023-04-23 05:54:35,202 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/a6ec29aa2c314a79838e2f72474ec403 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/a6ec29aa2c314a79838e2f72474ec403 2023-04-23 05:54:35,208 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/a6ec29aa2c314a79838e2f72474ec403, entries=150, sequenceid=255, filesize=11.8 K 2023-04-23 05:54:35,209 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for ba2bf059880117cc12577e45cc6d7717 in 581ms, sequenceid=255, compaction requested=false 2023-04-23 05:54:35,209 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:35,209 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:35,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:35,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:35,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,210 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:35,210 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:35,211 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,211 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:35,212 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,212 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:35,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:35,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7958ffae[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:35,213 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:35,213 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,214 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:35,215 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,215 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:35,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:35,219 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:35,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:35,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:35,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:35,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:35,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:35,222 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:35,222 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:35,222 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:35,222 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:35,222 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:35,224 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,224 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:35,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,224 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:35,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:35,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:35,224 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:35,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:35,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:35,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,229 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:35,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:35,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:35,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:35,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,229 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:35,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,229 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:35,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:35,330 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:35,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:35,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:35,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:35,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:35,333 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:35,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:35,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:35,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:35,334 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,334 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,335 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:35,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:35,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:35,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:35,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:35,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:35,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:35,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:35,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:35,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:35,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:35,337 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:35,337 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:35,338 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:54:35,338 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:35,338 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:35,338 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:35,338 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:35,338 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:35,338 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:35,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:35,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:35,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682229275413, max timestamp=1682229275423Num uniques 859; 2023-04-23 05:54:35,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682229275413, max timestamp=1682229275423Num uniques 859; 2023-04-23 05:54:35,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-23 05:54:35,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-23 05:54:35,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1682229275413, max timestamp=1682229275423Num uniques 859; 2023-04-23 05:54:35,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-23 05:54:35,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229335439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229335441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229335443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229335544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229335546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229335546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229335746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229335747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:35,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229335748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,757 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d4a024855253479a9603a2b8bac9dac3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d4a024855253479a9603a2b8bac9dac3 2023-04-23 05:54:35,766 WARN [DataStreamer for file /user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/8cd704a6e3af42fdb91ad4ee088b169e] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-23 05:54:35,767 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/8cd704a6e3af42fdb91ad4ee088b169e 2023-04-23 05:54:35,778 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/f8552de6c94642ca8d2b579ac9ade5fa 2023-04-23 05:54:35,790 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/435a967589e540108e0fb7a0fd8b5d2a 2023-04-23 05:54:35,796 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/8cd704a6e3af42fdb91ad4ee088b169e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/8cd704a6e3af42fdb91ad4ee088b169e 2023-04-23 05:54:35,801 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/8cd704a6e3af42fdb91ad4ee088b169e, entries=150, sequenceid=269, filesize=25.4 K 2023-04-23 05:54:35,806 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/f8552de6c94642ca8d2b579ac9ade5fa as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/f8552de6c94642ca8d2b579ac9ade5fa 2023-04-23 05:54:35,812 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/f8552de6c94642ca8d2b579ac9ade5fa, entries=150, sequenceid=269, filesize=11.9 K 2023-04-23 05:54:35,814 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/435a967589e540108e0fb7a0fd8b5d2a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/435a967589e540108e0fb7a0fd8b5d2a 2023-04-23 05:54:35,821 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/435a967589e540108e0fb7a0fd8b5d2a, entries=150, sequenceid=269, filesize=11.9 K 2023-04-23 05:54:35,822 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for ba2bf059880117cc12577e45cc6d7717 in 484ms, sequenceid=269, compaction requested=true 2023-04-23 05:54:35,822 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:35,822 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:35,822 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:35,822 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:35,822 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,822 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:35,822 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:35,825 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:35,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:35,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d01cab9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:35,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:35,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,833 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:35,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:35,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:35,834 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,834 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:35,835 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,835 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:35,835 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:35,835 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:35,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:35,838 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:35,838 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:35,838 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:35,838 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:35,838 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:35,840 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,840 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:35,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,840 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:35,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:35,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:35,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:35,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:35,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,841 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:35,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,843 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:35,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:35,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:35,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:35,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:35,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:35,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:35,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:35,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:35,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:35,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,847 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:35,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:35,847 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:35,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:35,947 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:35,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:35,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:35,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:35,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:35,951 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:35,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:35,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:35,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:35,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,952 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:35,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:35,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:35,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:35,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:35,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:35,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:35,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:35,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:35,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:35,954 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:35,954 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:35,954 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:35,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:35,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:35,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:35,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:35,955 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=134.18 KB heapSize=361.55 KB 2023-04-23 05:54:35,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:35,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:35,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:35,956 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:35,956 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:35,956 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:35,977 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042311cd52a9fb3d44d69ad2fe103fa271e3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042311cd52a9fb3d44d69ad2fe103fa271e3 2023-04-23 05:54:35,983 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=301, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/b9d3cf7f3c5b4baaadbd6345726b9969 2023-04-23 05:54:35,994 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/d26ae787cea641ac88942e0006c2f37c 2023-04-23 05:54:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:36,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:36,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229336062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229336063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229336064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229336166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229336167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229336167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229336369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229336370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229336371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,406 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/f06c8289543b4797a4e1febeeed18c18 2023-04-23 05:54:36,411 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/b9d3cf7f3c5b4baaadbd6345726b9969 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b9d3cf7f3c5b4baaadbd6345726b9969 2023-04-23 05:54:36,416 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b9d3cf7f3c5b4baaadbd6345726b9969, entries=150, sequenceid=301, filesize=25.4 K 2023-04-23 05:54:36,417 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/d26ae787cea641ac88942e0006c2f37c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d26ae787cea641ac88942e0006c2f37c 2023-04-23 05:54:36,422 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d26ae787cea641ac88942e0006c2f37c, entries=150, sequenceid=301, filesize=11.9 K 2023-04-23 05:54:36,423 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/f06c8289543b4797a4e1febeeed18c18 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f06c8289543b4797a4e1febeeed18c18 2023-04-23 05:54:36,427 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f06c8289543b4797a4e1febeeed18c18, entries=150, sequenceid=301, filesize=11.9 K 2023-04-23 05:54:36,428 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~361.50 KB/370176, currentSize=53.67 KB/54960 for ba2bf059880117cc12577e45cc6d7717 in 473ms, sequenceid=301, compaction requested=true 2023-04-23 05:54:36,428 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:36,428 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:36,428 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:36,428 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:36,428 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,428 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:36,428 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:36,430 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,430 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:36,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b258437[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:36,432 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:36,432 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,433 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:36,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:36,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:36,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:36,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:36,436 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,435 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,436 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:36,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:36,438 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:36,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:36,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:36,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:36,438 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:36,439 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,439 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,443 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:36,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:36,445 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:36,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:36,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:36,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,445 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:36,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,445 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:36,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:36,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:36,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:54:36,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:36,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:36,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:36,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:36,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:36,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:36,546 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:36,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:36,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:36,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:36,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:36,550 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:36,550 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:36,550 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:36,550 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:36,550 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,550 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,551 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:36,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:36,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:36,552 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,552 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:36,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:36,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:36,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:36,553 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:36,553 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:36,553 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:36,553 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:36,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:36,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229336554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,556 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:36,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@570689ec[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:36,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:36,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,559 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,559 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:36,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:36,559 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:36,559 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:36,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:36,560 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:36,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:36,563 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:36,563 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:36,563 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:36,563 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:36,563 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:36,567 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,567 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,567 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:36,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,569 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,571 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:36,571 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,571 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:36,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:36,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:36,571 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,571 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,571 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,571 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,571 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,571 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,571 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,571 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:36,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:36,571 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:36,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:36,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229336656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229336671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,672 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:36,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:36,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:36,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229336673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:36,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:36,675 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:36,675 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:36,675 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:36,675 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:36,676 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,676 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,677 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:36,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:36,677 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,677 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:36,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229336675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:36,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:36,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:36,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:36,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:36,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:36,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:36,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:36,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:36,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691db92d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:36,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:36,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,686 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:36,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:36,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:36,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:36,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:36,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:36,690 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:36,690 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:36,690 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:36,690 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:36,690 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:36,691 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,691 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,691 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,692 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,693 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:36,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:36,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:36,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:36,696 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:36,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:36,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,696 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:36,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,696 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:36,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,697 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:36,797 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:36,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:36,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:36,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:36,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:36,800 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:36,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:36,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:36,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:36,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,803 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:36,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:36,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,804 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:36,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:36,804 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:36,805 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:36,805 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:36,805 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:36,805 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:36,807 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:36,809 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:36,809 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4ae2b5[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:36,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:36,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:36,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:36,810 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,810 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:36,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:36,812 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,812 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:36,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:36,814 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:36,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:36,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:36,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:36,814 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:36,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,818 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,819 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:36,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:36,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:36,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:36,821 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:36,821 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:36,821 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:36,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:36,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:36,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229336859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,913 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d4dbe4af70784bbb97fec3ce9c37b20b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d4dbe4af70784bbb97fec3ce9c37b20b 2023-04-23 05:54:36,921 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:36,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:36,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:36,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:36,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:36,925 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:36,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:36,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:36,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:36,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,927 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,927 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:36,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:36,927 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,927 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:36,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:36,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:36,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:36,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:36,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:36,929 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:36,929 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:36,929 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:36,929 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:36,930 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,930 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,931 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,931 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:36,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc9d7f7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:36,932 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:36,932 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,934 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:36,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:36,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:36,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:36,936 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:36,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:36,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:36,938 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:36,939 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:36,939 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:36,939 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:36,939 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:36,940 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,940 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,940 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:36,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,940 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:36,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,942 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:36,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:36,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:36,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:36,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:36,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:36,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:36,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:36,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,949 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:36,949 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:36,949 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:36,949 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:36,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:36,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,049 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:37,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:37,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:37,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:37,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:37,052 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:37,052 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,052 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:37,052 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:37,053 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,053 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:37,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:37,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,055 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:37,056 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,056 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:37,056 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:37,056 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:37,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:37,058 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,058 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,058 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,058 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:37,059 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:37,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e636a4b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:37,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,060 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:37,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:37,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:37,061 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,061 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:37,063 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:37,063 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,063 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,063 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:37,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:37,065 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:37,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:37,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:37,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:37,066 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:37,067 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,067 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,067 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,067 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,069 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:37,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:37,071 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,071 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:37,071 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:37,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:37,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229337102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,108 DEBUG [Thread-818] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:54:37,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:37,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229337163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,171 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:37,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:37,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:37,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:37,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:37,174 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:37,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:37,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:37,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:37,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229337175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:37,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229337176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,180 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,180 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,181 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,181 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,181 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,181 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:37,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:37,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:37,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:37,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:37,182 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:37,182 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,182 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:37,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:37,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:37,183 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,183 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:37,183 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:37,183 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:37,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:37,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:37,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:37,186 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,186 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,186 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,186 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:37,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:37,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229337183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a336e7b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:37,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:37,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,188 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:37,189 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:37,189 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,189 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:37,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:37,192 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:37,192 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:37,192 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:37,193 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:37,193 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:37,194 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,194 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,194 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,194 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,196 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,198 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:37,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,199 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:37,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:37,200 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:37,200 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:37,300 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:37,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:37,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:37,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:37,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:37,303 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:37,303 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,303 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:37,303 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:37,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:37,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:37,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:37,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:37,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:37,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:37,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:37,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:37,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:37,307 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,307 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:37,307 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:37,307 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:37,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:37,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:37,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:37,309 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,309 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,309 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,309 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:37,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:37,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:37,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37516ac2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:37,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,312 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:37,312 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:37,312 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,312 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:37,315 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:37,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:37,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:37,318 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:37,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:37,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:37,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:37,318 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:37,319 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,319 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,319 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=313, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/b70fc015bd2f474ba61a6695e244648b 2023-04-23 05:54:37,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,319 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,320 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,322 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,329 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,329 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:37,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:37,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:37,329 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:37,330 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:37,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,430 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:37,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:37,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:37,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:37,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:37,433 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:37,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:37,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:37,434 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,434 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,436 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,436 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,436 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,436 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:37,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:37,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:37,437 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,437 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:37,437 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:37,437 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,437 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:37,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:37,439 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,439 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:37,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@148465b0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:37,440 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:37,441 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,442 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,442 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:37,442 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:37,442 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:37,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:37,443 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,443 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:37,444 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,444 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:37,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:37,446 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:37,446 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:37,446 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:37,446 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:37,446 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:37,448 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,448 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,448 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,448 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,451 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,453 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,453 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,453 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,453 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:37,453 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,454 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,454 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,454 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,453 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,453 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:37,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:37,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,454 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:37,454 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:37,554 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:37,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:37,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:37,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:37,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:37,558 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:37,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:37,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:37,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,560 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:37,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:37,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:37,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:37,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:37,562 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,562 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:37,562 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:37,562 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:37,564 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,564 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,564 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,564 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:37,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a06459[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:37,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:37,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:37,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:37,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:37,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:37,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,570 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:37,570 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,570 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:37,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:37,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:37,573 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:37,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:37,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:37,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:37,573 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:37,575 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,575 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,575 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,575 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,576 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,579 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,579 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,579 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:37,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:37,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:37,580 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,580 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,580 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,580 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,580 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,580 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,580 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,580 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:37,580 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:37,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:37,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36236 deadline: 1682229337668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,680 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:37,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:37,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:37,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:37,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:37,684 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:37,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:37,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:37,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,687 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,687 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,687 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:37,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:37,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:37,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:37,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:37,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:37,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:37,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:37,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:37,689 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,689 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:37,689 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:37,689 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:37,691 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,691 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,692 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:37,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f192a0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:37,693 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:37,693 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:37,695 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:37,695 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,695 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:37,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,696 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:37,696 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,696 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:37,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:37,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:37,701 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:37,701 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:37,701 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:37,701 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:37,701 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:37,704 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,704 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,705 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,706 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,711 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,711 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,711 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,711 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:37,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:37,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,712 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:37,712 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:37,711 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,711 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,712 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,712 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,712 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,712 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/3b3a31f9e6b74ec7b91fc2a2bb85e2e5 2023-04-23 05:54:37,812 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:37,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:37,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:37,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:37,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:37,816 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:37,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:37,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:37,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,817 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,818 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,818 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:37,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:37,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:37,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:37,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:37,819 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,819 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:37,819 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:37,819 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:37,819 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:37,820 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:37,824 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:37,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19d48955[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:37,825 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:37,825 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:37,827 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,827 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:37,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:37,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:37,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:37,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:37,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:37,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:37,831 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:37,831 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:37,831 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:37,831 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:37,831 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:37,833 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,833 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,833 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,834 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,835 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,837 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,837 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,837 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,837 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,837 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,837 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,837 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,837 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,837 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:37,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:37,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:37,837 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,837 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:37,837 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:37,938 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:37,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:37,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:37,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:37,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:37,941 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:37,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:37,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:37,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,942 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,942 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,943 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,943 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:37,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:37,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:37,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:37,944 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,944 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:37,945 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:37,945 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:37,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:37,946 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,946 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,947 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,947 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:37,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:37,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52c7c5d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:37,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,951 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,951 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:37,951 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:37,951 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:37,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:37,952 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,952 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,953 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,953 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:37,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:37,960 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:37,960 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:37,960 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:37,960 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:37,960 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:37,962 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,962 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,962 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:37,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:37,962 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:37,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:37,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,963 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:37,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:37,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:37,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:37,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:37,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:37,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:37,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:37,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:37,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:37,966 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:37,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:37,966 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:37,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:37,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:37,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,066 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:38,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:38,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:38,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:38,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:38,070 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:38,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:38,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:38,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,073 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,073 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,074 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,074 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:38,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:38,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:38,075 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,075 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:38,075 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:38,075 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:38,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:38,077 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,077 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,077 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,077 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:38,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:38,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef51323[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:38,079 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:38,079 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,080 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,080 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:38,080 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:38,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:38,080 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:38,082 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,082 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:38,082 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,082 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:38,084 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:38,084 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:38,084 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:38,084 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:38,084 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:38,086 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,086 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,087 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,087 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,088 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:38,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:38,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,091 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,090 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,091 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:38,091 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:38,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:38,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,091 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/1f2fe727f8e84762a05b34cd94a76edf 2023-04-23 05:54:38,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/b70fc015bd2f474ba61a6695e244648b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b70fc015bd2f474ba61a6695e244648b 2023-04-23 05:54:38,162 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b70fc015bd2f474ba61a6695e244648b, entries=200, sequenceid=313, filesize=32.3 K 2023-04-23 05:54:38,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/3b3a31f9e6b74ec7b91fc2a2bb85e2e5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3b3a31f9e6b74ec7b91fc2a2bb85e2e5 2023-04-23 05:54:38,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3b3a31f9e6b74ec7b91fc2a2bb85e2e5, entries=150, sequenceid=313, filesize=11.9 K 2023-04-23 05:54:38,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/1f2fe727f8e84762a05b34cd94a76edf as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/1f2fe727f8e84762a05b34cd94a76edf 2023-04-23 05:54:38,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/1f2fe727f8e84762a05b34cd94a76edf, entries=150, sequenceid=313, filesize=11.9 K 2023-04-23 05:54:38,173 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ba2bf059880117cc12577e45cc6d7717 in 1678ms, sequenceid=313, compaction requested=true 2023-04-23 05:54:38,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:38,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:38,174 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:38,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:38,174 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:38,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:38,175 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137346 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:38,176 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61006 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:38,176 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/A is initiating minor compaction (all files) 2023-04-23 05:54:38,176 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/B is initiating minor compaction (all files) 2023-04-23 05:54:38,176 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/A in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,176 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/B in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,176 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ed82e7b138314f7e868154868f720ad4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3525f395ba3b48d5a565b49df909c29a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/f8552de6c94642ca8d2b579ac9ade5fa, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d26ae787cea641ac88942e0006c2f37c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3b3a31f9e6b74ec7b91fc2a2bb85e2e5] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=59.6 K 2023-04-23 05:54:38,176 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc7083c72ed648aa8403541f4d9fb1e6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d95d2f7676194bbda2ee2b43cef99282, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/8cd704a6e3af42fdb91ad4ee088b169e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b9d3cf7f3c5b4baaadbd6345726b9969, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b70fc015bd2f474ba61a6695e244648b] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=134.1 K 2023-04-23 05:54:38,176 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ed82e7b138314f7e868154868f720ad4, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682229272959 2023-04-23 05:54:38,177 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting dc7083c72ed648aa8403541f4d9fb1e6, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682229272959 2023-04-23 05:54:38,177 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3525f395ba3b48d5a565b49df909c29a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682229274169 2023-04-23 05:54:38,177 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting d95d2f7676194bbda2ee2b43cef99282, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682229274169 2023-04-23 05:54:38,177 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f8552de6c94642ca8d2b579ac9ade5fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682229274782 2023-04-23 05:54:38,178 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 8cd704a6e3af42fdb91ad4ee088b169e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682229274782 2023-04-23 05:54:38,178 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b9d3cf7f3c5b4baaadbd6345726b9969, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682229275438 2023-04-23 05:54:38,178 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d26ae787cea641ac88942e0006c2f37c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682229275438 2023-04-23 05:54:38,179 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b70fc015bd2f474ba61a6695e244648b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682229276057 2023-04-23 05:54:38,179 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3b3a31f9e6b74ec7b91fc2a2bb85e2e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682229276057 2023-04-23 05:54:38,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:38,185 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-23 05:54:38,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:38,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:38,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:38,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:38,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:38,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:38,191 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:38,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:38,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:38,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:38,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:38,195 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:38,195 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,195 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:38,195 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:38,196 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,196 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,197 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#B#compaction#155 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:38,197 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,197 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,198 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,198 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:38,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:38,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229338198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:38,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229338199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:38,204 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:38,204 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,204 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:38,206 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,206 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:38,206 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#A#compaction#154 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:38,206 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:38,206 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229338203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:38,210 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,210 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,210 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,210 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:38,213 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40398ac3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:38,213 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,214 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/5ef138dcfe614e899a594c80b89f7fc7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5ef138dcfe614e899a594c80b89f7fc7 2023-04-23 05:54:38,218 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,218 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:38,218 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:38,218 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:38,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:38,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:38,219 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,221 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/B of ba2bf059880117cc12577e45cc6d7717 into 5ef138dcfe614e899a594c80b89f7fc7(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:38,221 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:38,221 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/B, priority=11, startTime=1682229278174; duration=0sec 2023-04-23 05:54:38,221 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:38,222 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:38,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:38,223 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:38,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:38,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:38,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:38,223 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:38,223 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61006 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:38,223 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/C is initiating minor compaction (all files) 2023-04-23 05:54:38,223 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/C in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,223 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/acdbb257c04f4d21a063be3bc0262cba, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/a6ec29aa2c314a79838e2f72474ec403, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/435a967589e540108e0fb7a0fd8b5d2a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f06c8289543b4797a4e1febeeed18c18, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/1f2fe727f8e84762a05b34cd94a76edf] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=59.6 K 2023-04-23 05:54:38,224 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting acdbb257c04f4d21a063be3bc0262cba, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1682229272959 2023-04-23 05:54:38,225 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a6ec29aa2c314a79838e2f72474ec403, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682229274169 2023-04-23 05:54:38,225 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 435a967589e540108e0fb7a0fd8b5d2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1682229274782 2023-04-23 05:54:38,225 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f06c8289543b4797a4e1febeeed18c18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1682229275438 2023-04-23 05:54:38,226 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,226 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,226 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1f2fe727f8e84762a05b34cd94a76edf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682229276057 2023-04-23 05:54:38,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,226 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,227 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:38,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:38,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:38,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,232 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,232 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,232 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:38,232 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:38,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,239 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#C#compaction#156 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:38,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229338304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229338304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229338308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,332 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:38,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:38,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:38,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:38,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:38,335 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:38,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:38,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:38,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,338 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,338 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,338 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,338 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:38,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:38,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:38,340 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,340 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:38,340 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:38,340 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:38,342 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:38,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d1a3db0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:38,343 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:38,343 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:38,345 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,345 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:38,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:38,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:38,346 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:38,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:38,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:38,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:38,349 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:38,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:38,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:38,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:38,349 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:38,351 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,351 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,351 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,353 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:38,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:38,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:38,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,356 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,356 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,356 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:38,356 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:38,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,456 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:38,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:38,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:38,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:38,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:38,459 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:38,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:38,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:38,460 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,460 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,461 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,461 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,461 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,461 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:38,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:38,462 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:38,462 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,462 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:38,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:38,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:38,463 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,463 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:38,463 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:38,463 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:38,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:38,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:38,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:38,464 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,464 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,464 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,465 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:38,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b3278da[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:38,466 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:38,466 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,468 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,468 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:38,468 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:38,468 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:38,468 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,468 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:38,469 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:38,469 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,469 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,469 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:38,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:38,471 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:38,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:38,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:38,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:38,472 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:38,473 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,473 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,473 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,473 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,475 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,477 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,477 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,477 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,477 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,477 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:38,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:38,477 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,477 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:38,477 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,477 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:38,477 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,478 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,477 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:38,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229338508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229338508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229338513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,578 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:38,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:38,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:38,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:38,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:38,581 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:38,581 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,581 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:38,581 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:38,582 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,582 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,584 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,584 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,584 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,584 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:38,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:38,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:38,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:38,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:38,585 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:38,585 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,585 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:38,586 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,586 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:38,586 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:38,586 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:38,586 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:38,587 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,588 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:38,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:38,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d081c94[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:38,589 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:38,590 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,591 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,591 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:38,591 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:38,591 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,591 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:38,591 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:38,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:38,593 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,593 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:38,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:38,595 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:38,595 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:38,595 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:38,595 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:38,595 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:38,600 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,600 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:38,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,600 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:38,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,600 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:38,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,602 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:38,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:38,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:38,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:38,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:38,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:38,604 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:38,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:38,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:38,604 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:38,610 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304233153c0b33a764b9f9324e6f8ec5ce837 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233153c0b33a764b9f9324e6f8ec5ce837 2023-04-23 05:54:38,617 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=334, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/242278b642664fd8a683c8e412029dfc 2023-04-23 05:54:38,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/b7c3d648505d4e0898003d22fcc0d022 2023-04-23 05:54:38,649 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/c0bb7dd20a1247e9896104f52537ca27 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/c0bb7dd20a1247e9896104f52537ca27 2023-04-23 05:54:38,655 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/C of ba2bf059880117cc12577e45cc6d7717 into c0bb7dd20a1247e9896104f52537ca27(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:38,655 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:38,655 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/C, priority=11, startTime=1682229278174; duration=0sec 2023-04-23 05:54:38,655 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:38,660 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/ed8c2f0041f34c02b1cdd112eb25da71 2023-04-23 05:54:38,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/242278b642664fd8a683c8e412029dfc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/242278b642664fd8a683c8e412029dfc 2023-04-23 05:54:38,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/242278b642664fd8a683c8e412029dfc, entries=200, sequenceid=334, filesize=32.3 K 2023-04-23 05:54:38,671 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/b7c3d648505d4e0898003d22fcc0d022 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/b7c3d648505d4e0898003d22fcc0d022 2023-04-23 05:54:38,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/b7c3d648505d4e0898003d22fcc0d022, entries=150, sequenceid=334, filesize=11.9 K 2023-04-23 05:54:38,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/ed8c2f0041f34c02b1cdd112eb25da71 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ed8c2f0041f34c02b1cdd112eb25da71 2023-04-23 05:54:38,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36236 deadline: 1682229338678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ed8c2f0041f34c02b1cdd112eb25da71, entries=150, sequenceid=334, filesize=11.9 K 2023-04-23 05:54:38,689 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for ba2bf059880117cc12577e45cc6d7717 in 504ms, sequenceid=334, compaction requested=false 2023-04-23 05:54:38,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:38,705 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:38,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:38,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:38,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:38,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:38,707 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:38,707 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:38,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:38,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,709 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:38,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:38,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:38,709 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,710 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:38,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:38,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:38,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:38,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:38,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:38,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:38,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:38,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:38,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:38,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:38,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:38,711 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:54:38,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:38,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:38,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:38,712 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:38,712 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:38,712 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:38,732 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423ebcb920453b84889bd366141566add18 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423ebcb920453b84889bd366141566add18 2023-04-23 05:54:38,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:38,813 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:38,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682229278825, max timestamp=1682229278839Num uniques 859; 2023-04-23 05:54:38,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-23 05:54:38,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229338837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229338844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229338845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229338942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229338946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:38,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:38,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229338950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,138 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/62307d4d0b2740d7b36ff661e8a8e351 2023-04-23 05:54:39,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:39,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229339145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:39,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229339148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:39,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229339153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,429 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/c0cfbe17b8a740eb8d7d62dd263e95fc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c0cfbe17b8a740eb8d7d62dd263e95fc 2023-04-23 05:54:39,435 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/A of ba2bf059880117cc12577e45cc6d7717 into c0cfbe17b8a740eb8d7d62dd263e95fc(size=26.0 K), total size for store is 58.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-23 05:54:39,435 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:39,435 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/A, priority=11, startTime=1682229278174; duration=1sec 2023-04-23 05:54:39,436 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:39,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:39,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229339451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:39,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229339451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:39,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229339458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,552 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/38902316882f4f6f8b132806e131b1ce 2023-04-23 05:54:39,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:39,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229339956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:39,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229339958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,964 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/26279c63d5eb44e28156bb3e481fc73f 2023-04-23 05:54:39,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:39,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229339965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,970 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/62307d4d0b2740d7b36ff661e8a8e351 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/62307d4d0b2740d7b36ff661e8a8e351 2023-04-23 05:54:39,974 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/62307d4d0b2740d7b36ff661e8a8e351, entries=150, sequenceid=347, filesize=25.4 K 2023-04-23 05:54:39,975 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/38902316882f4f6f8b132806e131b1ce as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/38902316882f4f6f8b132806e131b1ce 2023-04-23 05:54:39,980 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/38902316882f4f6f8b132806e131b1ce, entries=150, sequenceid=347, filesize=11.9 K 2023-04-23 05:54:39,980 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/26279c63d5eb44e28156bb3e481fc73f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/26279c63d5eb44e28156bb3e481fc73f 2023-04-23 05:54:39,984 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/26279c63d5eb44e28156bb3e481fc73f, entries=150, sequenceid=347, filesize=11.9 K 2023-04-23 05:54:39,985 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=123.00 KB/125950 for ba2bf059880117cc12577e45cc6d7717 in 1274ms, sequenceid=347, compaction requested=true 2023-04-23 05:54:39,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:39,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:39,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:39,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:39,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:39,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:39,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41ebe864[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:39,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:39,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:39,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:39,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:39,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:39,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:39,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:39,991 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,991 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:39,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:39,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:39,992 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:39,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:39,995 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:39,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:39,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:39,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:39,995 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:39,997 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:39,997 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:39,997 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:39,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:39,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:39,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:39,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:40,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:40,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:40,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:40,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:40,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:40,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:40,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:40,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:40,002 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:40,002 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:40,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:40,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:40,102 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:40,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:40,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:40,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:40,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:40,106 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:40,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:40,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:40,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:40,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:40,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:40,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,108 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:40,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:40,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:40,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:40,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:40,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:40,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:40,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:40,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:40,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:40,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:40,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:40,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:40,111 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:40,111 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:40,111 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=123.00 KB heapSize=357.03 KB 2023-04-23 05:54:40,111 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:40,111 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:40,111 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:40,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:40,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:40,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:40,527 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042366275d30489240dea063c030b3f8321c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042366275d30489240dea063c030b3f8321c 2023-04-23 05:54:40,532 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=373, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/106dc61813b2455d9c3710996274b8ff 2023-04-23 05:54:40,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:40,701 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:40,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:40,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36236 deadline: 1682229340723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:40,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36236 deadline: 1682229340826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,943 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/78bdcf66f9aa46f3b0ffced286c5a0c0 2023-04-23 05:54:40,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:40,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229340961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:40,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229340963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:40,968 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/97e6a2c253ef445392b3cd1191058c2d 2023-04-23 05:54:40,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229340967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,973 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/106dc61813b2455d9c3710996274b8ff as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/106dc61813b2455d9c3710996274b8ff 2023-04-23 05:54:40,978 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/106dc61813b2455d9c3710996274b8ff, entries=150, sequenceid=373, filesize=25.4 K 2023-04-23 05:54:40,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/78bdcf66f9aa46f3b0ffced286c5a0c0 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/78bdcf66f9aa46f3b0ffced286c5a0c0 2023-04-23 05:54:40,983 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/78bdcf66f9aa46f3b0ffced286c5a0c0, entries=150, sequenceid=373, filesize=11.9 K 2023-04-23 05:54:40,983 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/97e6a2c253ef445392b3cd1191058c2d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/97e6a2c253ef445392b3cd1191058c2d 2023-04-23 05:54:40,988 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/97e6a2c253ef445392b3cd1191058c2d, entries=150, sequenceid=373, filesize=11.9 K 2023-04-23 05:54:40,989 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~356.98 KB/365552, currentSize=53.67 KB/54960 for ba2bf059880117cc12577e45cc6d7717 in 878ms, sequenceid=373, compaction requested=true 2023-04-23 05:54:40,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:40,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:40,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:40,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:40,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:40,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:40,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:40,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:40,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:40,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:40,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d30ba54[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:40,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:40,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:40,997 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:40,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:40,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:40,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:40,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:40,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:40,999 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:40,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:40,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:41,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:41,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:41,002 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:41,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:41,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:41,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:41,002 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:41,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:41,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:41,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,013 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:41,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,013 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:41,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:41,032 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:54:41,032 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:41,032 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:41,032 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:41,032 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:41,032 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:41,032 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:41,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:41,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36236 deadline: 1682229341086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,113 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:41,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:41,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:41,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:41,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:41,116 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:41,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:41,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:41,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,118 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:41,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:41,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:41,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:41,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:41,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:41,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:41,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@654cfb80[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:41,123 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:41,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,126 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,126 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:41,126 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:41,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:41,126 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:41,127 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,127 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:41,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,127 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:41,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:41,130 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:41,130 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:41,130 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:41,130 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:41,130 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:41,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,134 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:41,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1682229341136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,139 DEBUG [Thread-818] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:54:41,139 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,139 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,139 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:41,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:41,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,140 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:41,140 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:41,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:41,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36236 deadline: 1682229341188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,240 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:41,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:41,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:41,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:41,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:41,243 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:41,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:41,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:41,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:41,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:41,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:41,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:41,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:41,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:41,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:41,248 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,248 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:41,248 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:41,248 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:41,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:41,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,249 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:41,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68ada701[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:41,251 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:41,251 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,252 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,252 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:41,252 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,252 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:41,252 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:41,253 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:41,255 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:41,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:41,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:41,257 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:41,257 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:41,257 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:41,257 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:41,257 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:41,258 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,259 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,259 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,259 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,260 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,262 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:41,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:41,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:41,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,263 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,263 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,263 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:41,263 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,263 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:41,263 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,363 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:41,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:41,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:41,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:41,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:41,366 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:41,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:41,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:41,367 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,367 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,368 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:41,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:41,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:41,370 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,370 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:41,370 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:41,370 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:41,370 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:41,372 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,373 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,373 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:41,374 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:41,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@769bcab8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:41,374 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,375 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:41,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:41,375 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,375 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:41,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:41,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:41,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,376 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:41,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:41,379 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:41,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:41,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:41,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:41,379 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:41,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,381 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,382 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,384 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,384 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,385 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,384 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:41,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:41,385 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,385 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,385 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,385 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:41,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:41,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,385 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:41,385 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,385 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,385 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:41,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36236 deadline: 1682229341394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,465 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423f55e92d169dd4f37bde06da40fca5b4e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f55e92d169dd4f37bde06da40fca5b4e 2023-04-23 05:54:41,470 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=385, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/1875713f3ebb424293814dc7a6c229fb 2023-04-23 05:54:41,485 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:41,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:41,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:41,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:41,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:41,488 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:41,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:41,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:41,489 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,489 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,490 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,490 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,490 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,490 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:41,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:41,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:41,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:41,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:41,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:41,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:41,494 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:41,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:41,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e112c25[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:41,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:41,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,498 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:41,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:41,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:41,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,499 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:41,506 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:41,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:41,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:41,508 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:41,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:41,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:41,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:41,508 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:41,510 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,510 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,510 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,510 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,512 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,518 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:41,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:41,519 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:41,519 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:41,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:41,619 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:41,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:41,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:41,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:41,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:41,623 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:41,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:41,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:41,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,626 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,626 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:41,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:41,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:41,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:41,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:41,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:41,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:41,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:41,629 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:41,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c83549c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:41,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:41,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:41,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:41,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:41,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:41,634 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:41,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:41,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:41,636 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:41,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:41,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:41,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:41,636 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:41,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,641 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:41,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:41,644 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:41,644 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:41,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:41,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:41,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36236 deadline: 1682229341698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,744 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:41,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:41,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:41,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:41,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:41,747 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:41,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:41,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:41,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:41,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:41,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:41,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:41,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:41,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:41,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:41,753 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,753 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,753 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,753 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:41,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19137da2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:41,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:41,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,756 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:41,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:41,758 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:41,758 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,758 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,758 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:41,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:41,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:41,761 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:41,761 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:41,761 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:41,761 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:41,761 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:41,762 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,762 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,762 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,766 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:41,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,766 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:41,866 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:41,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:41,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:41,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:41,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:41,870 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:41,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:41,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:41,871 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,871 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,873 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,874 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:41,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:41,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:41,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:41,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:41,875 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,875 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:41,875 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:41,875 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:41,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:41,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,877 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,877 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,877 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:41,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:41,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f5aa32[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:41,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:41,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,879 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:41,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:41,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:41,880 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,880 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:41,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:41,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/1f4d5c09461f49c0a17e528bcb69f12d 2023-04-23 05:54:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:41,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:41,914 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:41,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:41,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:41,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:41,914 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:41,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:41,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:41,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:41,918 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,919 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:41,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:41,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:41,921 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:41,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:41,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:41,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:41,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:41,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:41,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:41,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:41,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:41,922 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:41,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:41,922 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:41,922 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:42,022 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:42,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:42,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:42,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:42,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:42,025 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:42,025 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:42,025 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:42,025 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:42,026 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,026 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:42,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:42,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:42,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:42,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:42,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:42,028 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,028 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:42,029 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:42,029 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:42,034 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:42,035 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2692c82d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:42,035 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,037 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:42,037 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:42,037 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:42,037 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,037 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:42,038 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,038 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,038 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:42,038 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:42,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:42,040 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:42,040 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:42,040 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:42,040 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:42,040 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:42,041 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,041 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,042 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:42,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,042 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:42,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:42,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:42,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:42,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:42,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:42,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:42,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:42,054 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:42,054 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:42,154 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:42,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:42,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:42,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:42,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:42,159 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:42,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:42,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:42,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:42,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,161 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:42,161 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:42,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,162 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,162 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:42,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:42,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:42,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:42,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:42,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:42,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:42,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:42,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:42,163 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,163 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:42,164 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:42,164 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:42,165 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,165 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,166 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,166 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:42,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20334fd7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:42,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:42,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,168 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:42,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:42,168 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:42,168 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,168 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:42,171 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:42,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:42,174 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:42,174 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:42,174 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:42,174 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:42,174 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:42,175 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,175 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:42,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,175 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:42,176 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,178 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:42,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:42,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:42,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:42,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:42,180 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:42,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,180 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:42,180 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:42,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:42,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:42,181 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:42,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36236 deadline: 1682229342203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,280 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:42,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:42,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:42,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:42,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:42,284 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:42,284 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:42,284 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:42,284 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:42,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,286 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:42,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:42,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:42,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:42,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:42,288 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,288 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:42,288 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:42,288 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:42,291 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,291 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:42,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22cfd81e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:42,293 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:42,293 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,294 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,294 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:42,295 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:42,295 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:42,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:42,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:42,298 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:42,298 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:42,298 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:42,298 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:42,299 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:42,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,302 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,303 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,306 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:42,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:42,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:42,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:42,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:42,307 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:42,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:42,307 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:42,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:42,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:42,317 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/2ec66d3ff677459f8fd0d04e38563980 2023-04-23 05:54:42,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/1875713f3ebb424293814dc7a6c229fb as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/1875713f3ebb424293814dc7a6c229fb 2023-04-23 05:54:42,328 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/1875713f3ebb424293814dc7a6c229fb, entries=200, sequenceid=385, filesize=32.3 K 2023-04-23 05:54:42,329 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/1f4d5c09461f49c0a17e528bcb69f12d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/1f4d5c09461f49c0a17e528bcb69f12d 2023-04-23 05:54:42,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/1f4d5c09461f49c0a17e528bcb69f12d, entries=150, sequenceid=385, filesize=11.9 K 2023-04-23 05:54:42,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/2ec66d3ff677459f8fd0d04e38563980 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/2ec66d3ff677459f8fd0d04e38563980 2023-04-23 05:54:42,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/2ec66d3ff677459f8fd0d04e38563980, entries=150, sequenceid=385, filesize=11.9 K 2023-04-23 05:54:42,339 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ba2bf059880117cc12577e45cc6d7717 in 1308ms, sequenceid=385, compaction requested=true 2023-04-23 05:54:42,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:42,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:42,339 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:42,339 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:42,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:42,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:42,341 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 144856 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:42,341 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:42,341 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/A is initiating minor compaction (all files) 2023-04-23 05:54:42,342 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/B is initiating minor compaction (all files) 2023-04-23 05:54:42,342 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/A in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,342 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/B in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,342 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c0cfbe17b8a740eb8d7d62dd263e95fc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/242278b642664fd8a683c8e412029dfc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/62307d4d0b2740d7b36ff661e8a8e351, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/106dc61813b2455d9c3710996274b8ff, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/1875713f3ebb424293814dc7a6c229fb] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=141.5 K 2023-04-23 05:54:42,342 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5ef138dcfe614e899a594c80b89f7fc7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/b7c3d648505d4e0898003d22fcc0d022, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/38902316882f4f6f8b132806e131b1ce, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/78bdcf66f9aa46f3b0ffced286c5a0c0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/1f4d5c09461f49c0a17e528bcb69f12d] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=60.0 K 2023-04-23 05:54:42,342 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c0cfbe17b8a740eb8d7d62dd263e95fc, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682229276057 2023-04-23 05:54:42,342 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 5ef138dcfe614e899a594c80b89f7fc7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682229276057 2023-04-23 05:54:42,343 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 242278b642664fd8a683c8e412029dfc, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1682229276540 2023-04-23 05:54:42,343 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b7c3d648505d4e0898003d22fcc0d022, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1682229276540 2023-04-23 05:54:42,343 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 62307d4d0b2740d7b36ff661e8a8e351, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1682229278193 2023-04-23 05:54:42,343 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 38902316882f4f6f8b132806e131b1ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1682229278193 2023-04-23 05:54:42,344 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 106dc61813b2455d9c3710996274b8ff, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682229278842 2023-04-23 05:54:42,344 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 78bdcf66f9aa46f3b0ffced286c5a0c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682229278842 2023-04-23 05:54:42,345 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 1875713f3ebb424293814dc7a6c229fb, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682229280702 2023-04-23 05:54:42,345 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1f4d5c09461f49c0a17e528bcb69f12d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682229280709 2023-04-23 05:54:42,358 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#B#compaction#169 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:42,360 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#A#compaction#168 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:42,407 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:42,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:42,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:42,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:42,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:42,410 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:42,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:42,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:42,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:42,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:42,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:42,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:42,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:42,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:42,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:42,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:42,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:42,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:42,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:42,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:42,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:42,414 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,414 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:42,414 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:54:42,415 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:42,415 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:42,415 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:42,415 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:42,415 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:42,415 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:42,442 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423472e0428af944094baf676fdaddff8f1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423472e0428af944094baf676fdaddff8f1 2023-04-23 05:54:42,781 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/731ae851000f4eee93e7f88b2ce94147 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/731ae851000f4eee93e7f88b2ce94147 2023-04-23 05:54:42,787 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/B of ba2bf059880117cc12577e45cc6d7717 into 731ae851000f4eee93e7f88b2ce94147(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:42,787 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:42,787 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/B, priority=11, startTime=1682229282339; duration=0sec 2023-04-23 05:54:42,787 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:42,787 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:42,790 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61476 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:42,790 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/C is initiating minor compaction (all files) 2023-04-23 05:54:42,790 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/C in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,790 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/c0bb7dd20a1247e9896104f52537ca27, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ed8c2f0041f34c02b1cdd112eb25da71, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/26279c63d5eb44e28156bb3e481fc73f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/97e6a2c253ef445392b3cd1191058c2d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/2ec66d3ff677459f8fd0d04e38563980] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=60.0 K 2023-04-23 05:54:42,790 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c0bb7dd20a1247e9896104f52537ca27, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1682229276057 2023-04-23 05:54:42,791 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ed8c2f0041f34c02b1cdd112eb25da71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1682229276540 2023-04-23 05:54:42,791 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 26279c63d5eb44e28156bb3e481fc73f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1682229278193 2023-04-23 05:54:42,792 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 97e6a2c253ef445392b3cd1191058c2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1682229278842 2023-04-23 05:54:42,792 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2ec66d3ff677459f8fd0d04e38563980, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682229280709 2023-04-23 05:54:42,804 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#C#compaction#171 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:42,814 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/4dc2382b83224a5bacd18bae2ac455b7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/4dc2382b83224a5bacd18bae2ac455b7 2023-04-23 05:54:42,819 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/C of ba2bf059880117cc12577e45cc6d7717 into 4dc2382b83224a5bacd18bae2ac455b7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:42,819 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:42,819 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/C, priority=11, startTime=1682229282339; duration=0sec 2023-04-23 05:54:42,819 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:42,854 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/b0b4d4e6338c41d990685e5845e203ed 2023-04-23 05:54:42,866 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/43a3abf5e28b4085b79587d1139aa34a 2023-04-23 05:54:42,879 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/ca0a09164f8443dc95d7bc9851080127 2023-04-23 05:54:42,884 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/b0b4d4e6338c41d990685e5845e203ed as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b0b4d4e6338c41d990685e5845e203ed 2023-04-23 05:54:42,888 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b0b4d4e6338c41d990685e5845e203ed, entries=150, sequenceid=405, filesize=25.4 K 2023-04-23 05:54:42,889 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/43a3abf5e28b4085b79587d1139aa34a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/43a3abf5e28b4085b79587d1139aa34a 2023-04-23 05:54:42,894 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/43a3abf5e28b4085b79587d1139aa34a, entries=150, sequenceid=405, filesize=11.9 K 2023-04-23 05:54:42,895 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/ca0a09164f8443dc95d7bc9851080127 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ca0a09164f8443dc95d7bc9851080127 2023-04-23 05:54:42,901 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ca0a09164f8443dc95d7bc9851080127, entries=150, sequenceid=405, filesize=11.9 K 2023-04-23 05:54:42,902 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for ba2bf059880117cc12577e45cc6d7717 in 488ms, sequenceid=405, compaction requested=false 2023-04-23 05:54:42,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:42,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:42,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:42,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:42,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:42,907 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:42,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,910 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,910 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:42,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e77424f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:42,912 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:42,912 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,916 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:42,917 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,917 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:42,918 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:42,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:42,922 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:42,922 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:42,922 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:42,922 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:42,922 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:42,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:42,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:42,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:42,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:42,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:42,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:42,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:42,928 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:42,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:42,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:42,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,928 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:42,928 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:42,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:42,929 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:42,929 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,929 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:42,929 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:43,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:54:43,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:43,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:43,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:43,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:43,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:43,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:43,029 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:43,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:43,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:43,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:43,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:43,034 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:43,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:43,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:43,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:43,035 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,035 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:43,038 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:43,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:43,038 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:43,039 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,039 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:43,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:43,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:43,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:43,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:43,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:43,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:43,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:43,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:43,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:43,041 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:43,041 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:43,041 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:43,041 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:43,043 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423847c6749fa314f97a75cbe8fffcd4442 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423847c6749fa314f97a75cbe8fffcd4442 2023-04-23 05:54:43,043 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,043 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:43,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:43,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:43,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:43,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e94360c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:43,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:43,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:43,047 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:43,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:43,047 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:43,047 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:43,049 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:43,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:43,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:43,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:43,053 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:43,053 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:43,053 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:43,053 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:43,053 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:43,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229283038, max timestamp=1682229283054Num uniques 859; 2023-04-23 05:54:43,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:43,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229283038, max timestamp=1682229283054Num uniques 859; 2023-04-23 05:54:43,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229283038, max timestamp=1682229283054Num uniques 859; 2023-04-23 05:54:43,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:43,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:54:43,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:43,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,058 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:43,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:43,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:43,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:43,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:43,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:43,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:43,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:43,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:43,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:43,062 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:43,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:43,063 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:43,063 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:43,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:43,075 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=417, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/d1bfccc6e46e4ad59529e90a933c0971 2023-04-23 05:54:43,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229343083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229343085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229343085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/76d9141ff5544756ba6a525bcba1a9c5 2023-04-23 05:54:43,111 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=417 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/023321e4c3e24997b5e2d9324e0499c5 2023-04-23 05:54:43,117 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/d1bfccc6e46e4ad59529e90a933c0971 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d1bfccc6e46e4ad59529e90a933c0971 2023-04-23 05:54:43,122 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d1bfccc6e46e4ad59529e90a933c0971, entries=350, sequenceid=417, filesize=52.9 K 2023-04-23 05:54:43,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/76d9141ff5544756ba6a525bcba1a9c5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/76d9141ff5544756ba6a525bcba1a9c5 2023-04-23 05:54:43,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/76d9141ff5544756ba6a525bcba1a9c5, entries=150, sequenceid=417, filesize=11.9 K 2023-04-23 05:54:43,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/023321e4c3e24997b5e2d9324e0499c5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/023321e4c3e24997b5e2d9324e0499c5 2023-04-23 05:54:43,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/023321e4c3e24997b5e2d9324e0499c5, entries=150, sequenceid=417, filesize=11.9 K 2023-04-23 05:54:43,140 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ba2bf059880117cc12577e45cc6d7717 in 122ms, sequenceid=417, compaction requested=true 2023-04-23 05:54:43,141 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:43,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:43,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:43,141 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-23 05:54:43,141 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 5 compacting, 2 eligible, 16 blocking 2023-04-23 05:54:43,142 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-23 05:54:43,142 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-23 05:54:43,142 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. because compaction request was cancelled 2023-04-23 05:54:43,142 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:43,143 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:43,143 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/B is initiating minor compaction (all files) 2023-04-23 05:54:43,143 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/B in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:43,143 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/731ae851000f4eee93e7f88b2ce94147, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/43a3abf5e28b4085b79587d1139aa34a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/76d9141ff5544756ba6a525bcba1a9c5] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=36.4 K 2023-04-23 05:54:43,144 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 731ae851000f4eee93e7f88b2ce94147, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682229280709 2023-04-23 05:54:43,144 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 43a3abf5e28b4085b79587d1139aa34a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682229281072 2023-04-23 05:54:43,145 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 76d9141ff5544756ba6a525bcba1a9c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1682229283015 2023-04-23 05:54:43,152 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#B#compaction#177 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:43,163 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:43,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:43,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:43,166 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/348b46a46df04894851ded2885165ca7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/348b46a46df04894851ded2885165ca7 2023-04-23 05:54:43,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:43,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:43,167 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:43,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:43,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:43,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:43,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:43,169 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,169 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:43,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:43,169 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,169 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:43,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:43,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:43,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:43,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:43,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:43,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:43,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:43,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:43,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:43,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:43,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:43,171 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:43,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:43,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. started... 2023-04-23 05:54:43,171 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-23 05:54:43,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:43,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:43,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:43,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:43,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:43,172 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:43,172 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/B of ba2bf059880117cc12577e45cc6d7717 into 348b46a46df04894851ded2885165ca7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:43,172 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:43,172 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/B, priority=13, startTime=1682229283141; duration=0sec 2023-04-23 05:54:43,172 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:43,172 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:43,174 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:43,174 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/C is initiating minor compaction (all files) 2023-04-23 05:54:43,174 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/C in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:43,174 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/4dc2382b83224a5bacd18bae2ac455b7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ca0a09164f8443dc95d7bc9851080127, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/023321e4c3e24997b5e2d9324e0499c5] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=36.4 K 2023-04-23 05:54:43,175 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 4dc2382b83224a5bacd18bae2ac455b7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682229280709 2023-04-23 05:54:43,175 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ca0a09164f8443dc95d7bc9851080127, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682229281072 2023-04-23 05:54:43,175 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 023321e4c3e24997b5e2d9324e0499c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1682229283015 2023-04-23 05:54:43,185 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#C#compaction#178 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:54:43,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:43,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. as already flushing 2023-04-23 05:54:43,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229343198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229343199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229343200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36236 deadline: 1682229343218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229343301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229343301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229343308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229343503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229343504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229343512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,608 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/d7dd17d0a6e94f3692300dc27f466b52 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d7dd17d0a6e94f3692300dc27f466b52 2023-04-23 05:54:43,613 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/3446f2c6b14e415bba3a0b94e6ae7741 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/3446f2c6b14e415bba3a0b94e6ae7741 2023-04-23 05:54:43,615 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423c3f9c8475fbc45288f17cfdbc6479bc3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c3f9c8475fbc45288f17cfdbc6479bc3 2023-04-23 05:54:43,616 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/C of ba2bf059880117cc12577e45cc6d7717 into d7dd17d0a6e94f3692300dc27f466b52(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:43,616 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:43,616 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/C, priority=13, startTime=1682229283141; duration=0sec 2023-04-23 05:54:43,616 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:43,623 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/A of ba2bf059880117cc12577e45cc6d7717 into 3446f2c6b14e415bba3a0b94e6ae7741(size=26.1 K), total size for store is 104.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-23 05:54:43,623 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:43,623 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/A, priority=11, startTime=1682229282339; duration=1sec 2023-04-23 05:54:43,623 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:43,631 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=452, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/63f3cd16999b460387a007ae5f7bf108 2023-04-23 05:54:43,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229343807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229343809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:43,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:43,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229343817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,046 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/c6d14693933f40f799aab1c7a6325034 2023-04-23 05:54:44,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:44,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1682229344309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:44,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36268 deadline: 1682229344315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:44,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36260 deadline: 1682229344322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,460 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/0b84a9982da6454c8009b50934b9d87a 2023-04-23 05:54:44,466 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/63f3cd16999b460387a007ae5f7bf108 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/63f3cd16999b460387a007ae5f7bf108 2023-04-23 05:54:44,472 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/63f3cd16999b460387a007ae5f7bf108, entries=150, sequenceid=452, filesize=25.4 K 2023-04-23 05:54:44,473 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/c6d14693933f40f799aab1c7a6325034 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/c6d14693933f40f799aab1c7a6325034 2023-04-23 05:54:44,478 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/c6d14693933f40f799aab1c7a6325034, entries=150, sequenceid=452, filesize=11.9 K 2023-04-23 05:54:44,478 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/0b84a9982da6454c8009b50934b9d87a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/0b84a9982da6454c8009b50934b9d87a 2023-04-23 05:54:44,482 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/0b84a9982da6454c8009b50934b9d87a, entries=150, sequenceid=452, filesize=11.9 K 2023-04-23 05:54:44,483 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for ba2bf059880117cc12577e45cc6d7717 in 1312ms, sequenceid=452, compaction requested=true 2023-04-23 05:54:44,484 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:44,484 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:44,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:44,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:44,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:44,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:44,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:44,485 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:44,486 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:44,486 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:44,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:44,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34638dfe[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:44,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:44,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:44,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:44,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:44,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:44,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:44,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:44,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:44,499 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:44,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:44,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:44,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:44,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:44,503 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:44,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:44,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:44,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:44,503 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:44,506 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:44,506 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:44,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:44,506 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:44,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:44,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:44,512 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,513 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:44,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,550 DEBUG [Thread-825] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b416c44 to 127.0.0.1:63802 2023-04-23 05:54:44,550 DEBUG [Thread-825] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:44,550 DEBUG [Thread-827] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x204b5880 to 127.0.0.1:63802 2023-04-23 05:54:44,550 DEBUG [Thread-827] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:44,551 DEBUG [Thread-823] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2668d41d to 127.0.0.1:63802 2023-04-23 05:54:44,551 DEBUG [Thread-823] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:44,552 DEBUG [Thread-829] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41173aec to 127.0.0.1:63802 2023-04-23 05:54:44,552 DEBUG [Thread-829] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:44,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:44,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:44,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:44,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:44,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:44,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:44,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:44,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:44,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:44,619 DEBUG [Thread-822] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:44,619 DEBUG [Thread-822] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:44,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:44,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:44,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:44,620 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:44,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:44,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:44,719 DEBUG [Thread-822] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:44,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:45,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:45,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:54:45,234 DEBUG [Thread-816] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18496e17 to 127.0.0.1:63802 2023-04-23 05:54:45,234 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:45,235 DEBUG [Thread-816] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:45,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:45,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:45,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:45,235 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:45,235 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:45,321 DEBUG [Thread-820] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e0a3dc to 127.0.0.1:63802 2023-04-23 05:54:45,321 DEBUG [Thread-820] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:45,323 DEBUG [Thread-814] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52d98fed to 127.0.0.1:63802 2023-04-23 05:54:45,323 DEBUG [Thread-814] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:45,333 DEBUG [Thread-812] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63ebf1e9 to 127.0.0.1:63802 2023-04-23 05:54:45,333 DEBUG [Thread-812] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:45,651 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423a5d3a146f72e459ba0743302e5497d1a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a5d3a146f72e459ba0743302e5497d1a 2023-04-23 05:54:46,056 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=466, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/5cb0902dc7404ca3bb8db9ea29410ec4 2023-04-23 05:54:46,467 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/7de75b5c78834684a86a5a2c031fa5aa 2023-04-23 05:54:46,878 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/be464d4419794cc88c84a83227aca552 2023-04-23 05:54:46,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/5cb0902dc7404ca3bb8db9ea29410ec4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/5cb0902dc7404ca3bb8db9ea29410ec4 2023-04-23 05:54:46,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/5cb0902dc7404ca3bb8db9ea29410ec4, entries=150, sequenceid=466, filesize=25.4 K 2023-04-23 05:54:46,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/7de75b5c78834684a86a5a2c031fa5aa as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7de75b5c78834684a86a5a2c031fa5aa 2023-04-23 05:54:46,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7de75b5c78834684a86a5a2c031fa5aa, entries=150, sequenceid=466, filesize=11.9 K 2023-04-23 05:54:46,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/be464d4419794cc88c84a83227aca552 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/be464d4419794cc88c84a83227aca552 2023-04-23 05:54:46,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/be464d4419794cc88c84a83227aca552, entries=150, sequenceid=466, filesize=11.9 K 2023-04-23 05:54:46,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=20.13 KB/20610 for ba2bf059880117cc12577e45cc6d7717 in 1661ms, sequenceid=466, compaction requested=true 2023-04-23 05:54:46,896 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:46,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:46,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:46,896 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:54:46,896 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:46,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:46,897 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:46,897 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/B is initiating minor compaction (all files) 2023-04-23 05:54:46,897 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/B in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:46,897 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/348b46a46df04894851ded2885165ca7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/c6d14693933f40f799aab1c7a6325034, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7de75b5c78834684a86a5a2c031fa5aa] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=36.5 K 2023-04-23 05:54:46,897 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 159114 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:54:46,898 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/A is initiating minor compaction (all files) 2023-04-23 05:54:46,898 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/A in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:46,898 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/3446f2c6b14e415bba3a0b94e6ae7741, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b0b4d4e6338c41d990685e5845e203ed, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d1bfccc6e46e4ad59529e90a933c0971, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/63f3cd16999b460387a007ae5f7bf108, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/5cb0902dc7404ca3bb8db9ea29410ec4] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=155.4 K 2023-04-23 05:54:46,898 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 348b46a46df04894851ded2885165ca7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1682229283015 2023-04-23 05:54:46,898 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 3446f2c6b14e415bba3a0b94e6ae7741, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1682229280709 2023-04-23 05:54:46,898 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c6d14693933f40f799aab1c7a6325034, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682229283080 2023-04-23 05:54:46,898 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b0b4d4e6338c41d990685e5845e203ed, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1682229281072 2023-04-23 05:54:46,899 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7de75b5c78834684a86a5a2c031fa5aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1682229283197 2023-04-23 05:54:46,899 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting d1bfccc6e46e4ad59529e90a933c0971, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1682229282997 2023-04-23 05:54:46,899 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 63f3cd16999b460387a007ae5f7bf108, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682229283080 2023-04-23 05:54:46,900 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 5cb0902dc7404ca3bb8db9ea29410ec4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1682229283197 2023-04-23 05:54:46,905 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#B#compaction#185 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:54:46,913 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#A#compaction#186 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:54:47,318 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/065257578fbb4933be5bb7cfd001cc9d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/065257578fbb4933be5bb7cfd001cc9d 2023-04-23 05:54:47,326 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/B of ba2bf059880117cc12577e45cc6d7717 into 065257578fbb4933be5bb7cfd001cc9d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:47,326 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:47,326 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/B, priority=13, startTime=1682229286896; duration=0sec 2023-04-23 05:54:47,327 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:47,327 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:47,328 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:47,328 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): ba2bf059880117cc12577e45cc6d7717/C is initiating minor compaction (all files) 2023-04-23 05:54:47,328 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ba2bf059880117cc12577e45cc6d7717/C in TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:47,328 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d7dd17d0a6e94f3692300dc27f466b52, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/0b84a9982da6454c8009b50934b9d87a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/be464d4419794cc88c84a83227aca552] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp, totalSize=36.5 K 2023-04-23 05:54:47,329 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d7dd17d0a6e94f3692300dc27f466b52, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=417, earliestPutTs=1682229283015 2023-04-23 05:54:47,329 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 0b84a9982da6454c8009b50934b9d87a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1682229283080 2023-04-23 05:54:47,330 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting be464d4419794cc88c84a83227aca552, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1682229283197 2023-04-23 05:54:47,337 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): ba2bf059880117cc12577e45cc6d7717#C#compaction#187 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:54:47,747 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/754c5e966bc442ffae4897d828493f66 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/754c5e966bc442ffae4897d828493f66 2023-04-23 05:54:47,751 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/C of ba2bf059880117cc12577e45cc6d7717 into 754c5e966bc442ffae4897d828493f66(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:47,751 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:47,751 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/C, priority=13, startTime=1682229286896; duration=0sec 2023-04-23 05:54:47,751 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:48,138 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/240ff99259a6418b8029e63bb80a3bde as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/240ff99259a6418b8029e63bb80a3bde 2023-04-23 05:54:48,142 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ba2bf059880117cc12577e45cc6d7717/A of ba2bf059880117cc12577e45cc6d7717 into 240ff99259a6418b8029e63bb80a3bde(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-23 05:54:48,142 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:48,142 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717., storeName=ba2bf059880117cc12577e45cc6d7717/A, priority=11, startTime=1682229286896; duration=1sec 2023-04-23 05:54:48,142 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:51,212 DEBUG [Thread-818] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25441895 to 127.0.0.1:63802 2023-04-23 05:54:51,212 DEBUG [Thread-818] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 93 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 79 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 4920 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 4853 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 2166 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 6498 rows 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 2167 2023-04-23 05:54:51,212 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 6498 rows 2023-04-23 05:54:51,213 INFO [Listener at localhost/41845] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-23 05:54:51,213 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ab850e7 to 127.0.0.1:63802 2023-04-23 05:54:51,213 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:54:51,215 INFO [Listener at localhost/41845] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-23 05:54:51,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-23 05:54:51,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:51,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-23 05:54:51,224 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229291224"}]},"ts":"1682229291224"} 2023-04-23 05:54:51,225 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-23 05:54:51,239 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-23 05:54:51,240 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, UNASSIGN}] 2023-04-23 05:54:51,241 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, UNASSIGN 2023-04-23 05:54:51,242 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=ba2bf059880117cc12577e45cc6d7717, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:51,242 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229291242"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229291242"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229291242"}]},"ts":"1682229291242"} 2023-04-23 05:54:51,243 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:54:51,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-23 05:54:51,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ba2bf059880117cc12577e45cc6d7717, disabling compactions & flushes 2023-04-23 05:54:51,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. after waiting 0 ms 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:51,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ba2bf059880117cc12577e45cc6d7717 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=A 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=B 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ba2bf059880117cc12577e45cc6d7717, store=C 2023-04-23 05:54:51,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:51,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-23 05:54:51,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423327eb7f912314dd39a4fdae2d0ad817a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423327eb7f912314dd39a4fdae2d0ad817a 2023-04-23 05:54:51,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-23 05:54:52,214 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=476, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/c7059ac676164e8a94359869405ee158 2023-04-23 05:54:52,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-23 05:54:52,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/0ebe85af7485471891c6d4eb44d4a654 2023-04-23 05:54:53,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/45af3f4b905d4cfd966634af71879849 2023-04-23 05:54:53,040 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/A/c7059ac676164e8a94359869405ee158 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c7059ac676164e8a94359869405ee158 2023-04-23 05:54:53,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c7059ac676164e8a94359869405ee158, entries=150, sequenceid=476, filesize=25.4 K 2023-04-23 05:54:53,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/B/0ebe85af7485471891c6d4eb44d4a654 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/0ebe85af7485471891c6d4eb44d4a654 2023-04-23 05:54:53,049 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/0ebe85af7485471891c6d4eb44d4a654, entries=150, sequenceid=476, filesize=11.9 K 2023-04-23 05:54:53,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/.tmp/C/45af3f4b905d4cfd966634af71879849 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/45af3f4b905d4cfd966634af71879849 2023-04-23 05:54:53,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/45af3f4b905d4cfd966634af71879849, entries=150, sequenceid=476, filesize=11.9 K 2023-04-23 05:54:53,055 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for ba2bf059880117cc12577e45cc6d7717 in 1660ms, sequenceid=476, compaction requested=false 2023-04-23 05:54:53,074 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/94f410714b734894a875311bcd673f2b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/59b420bb172946c3aff7ff8846259324, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/194eedbf220e4d72a03b11188a3961ca, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/6605aee4310f4729bc4647563d61c9b2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d3b2d635e945402b970e781446366a1c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/ffc58ff0646e4a0da5a522000417fd63, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc1bd79878d4466eb2c9b1edc59580ff, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/05b11430917d46bb914a549e774bb5e9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c9ef2589658b4aefb2c230a18abf0435, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/75d7e5a5629c48f99b68e0a500aad549, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/7b78ceb645f144ff8bdcb06d4029482b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc7083c72ed648aa8403541f4d9fb1e6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d95d2f7676194bbda2ee2b43cef99282, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/8cd704a6e3af42fdb91ad4ee088b169e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b9d3cf7f3c5b4baaadbd6345726b9969, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b70fc015bd2f474ba61a6695e244648b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c0cfbe17b8a740eb8d7d62dd263e95fc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/242278b642664fd8a683c8e412029dfc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/62307d4d0b2740d7b36ff661e8a8e351, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/106dc61813b2455d9c3710996274b8ff, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/1875713f3ebb424293814dc7a6c229fb, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/3446f2c6b14e415bba3a0b94e6ae7741, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b0b4d4e6338c41d990685e5845e203ed, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d1bfccc6e46e4ad59529e90a933c0971, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/63f3cd16999b460387a007ae5f7bf108, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/5cb0902dc7404ca3bb8db9ea29410ec4] to archive 2023-04-23 05:54:53,075 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:54:53,077 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/94f410714b734894a875311bcd673f2b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/94f410714b734894a875311bcd673f2b 2023-04-23 05:54:53,078 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/59b420bb172946c3aff7ff8846259324 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/59b420bb172946c3aff7ff8846259324 2023-04-23 05:54:53,079 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/194eedbf220e4d72a03b11188a3961ca to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/194eedbf220e4d72a03b11188a3961ca 2023-04-23 05:54:53,080 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/6605aee4310f4729bc4647563d61c9b2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/6605aee4310f4729bc4647563d61c9b2 2023-04-23 05:54:53,081 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d3b2d635e945402b970e781446366a1c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d3b2d635e945402b970e781446366a1c 2023-04-23 05:54:53,082 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/ffc58ff0646e4a0da5a522000417fd63 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/ffc58ff0646e4a0da5a522000417fd63 2023-04-23 05:54:53,083 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc1bd79878d4466eb2c9b1edc59580ff to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc1bd79878d4466eb2c9b1edc59580ff 2023-04-23 05:54:53,084 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/05b11430917d46bb914a549e774bb5e9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/05b11430917d46bb914a549e774bb5e9 2023-04-23 05:54:53,085 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c9ef2589658b4aefb2c230a18abf0435 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c9ef2589658b4aefb2c230a18abf0435 2023-04-23 05:54:53,086 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/75d7e5a5629c48f99b68e0a500aad549 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/75d7e5a5629c48f99b68e0a500aad549 2023-04-23 05:54:53,087 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/7b78ceb645f144ff8bdcb06d4029482b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/7b78ceb645f144ff8bdcb06d4029482b 2023-04-23 05:54:53,088 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc7083c72ed648aa8403541f4d9fb1e6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/dc7083c72ed648aa8403541f4d9fb1e6 2023-04-23 05:54:53,089 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d95d2f7676194bbda2ee2b43cef99282 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d95d2f7676194bbda2ee2b43cef99282 2023-04-23 05:54:53,090 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/8cd704a6e3af42fdb91ad4ee088b169e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/8cd704a6e3af42fdb91ad4ee088b169e 2023-04-23 05:54:53,091 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b9d3cf7f3c5b4baaadbd6345726b9969 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b9d3cf7f3c5b4baaadbd6345726b9969 2023-04-23 05:54:53,092 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b70fc015bd2f474ba61a6695e244648b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b70fc015bd2f474ba61a6695e244648b 2023-04-23 05:54:53,093 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c0cfbe17b8a740eb8d7d62dd263e95fc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c0cfbe17b8a740eb8d7d62dd263e95fc 2023-04-23 05:54:53,094 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/242278b642664fd8a683c8e412029dfc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/242278b642664fd8a683c8e412029dfc 2023-04-23 05:54:53,095 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/62307d4d0b2740d7b36ff661e8a8e351 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/62307d4d0b2740d7b36ff661e8a8e351 2023-04-23 05:54:53,096 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/106dc61813b2455d9c3710996274b8ff to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/106dc61813b2455d9c3710996274b8ff 2023-04-23 05:54:53,097 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/1875713f3ebb424293814dc7a6c229fb to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/1875713f3ebb424293814dc7a6c229fb 2023-04-23 05:54:53,098 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/3446f2c6b14e415bba3a0b94e6ae7741 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/3446f2c6b14e415bba3a0b94e6ae7741 2023-04-23 05:54:53,099 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b0b4d4e6338c41d990685e5845e203ed to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/b0b4d4e6338c41d990685e5845e203ed 2023-04-23 05:54:53,100 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d1bfccc6e46e4ad59529e90a933c0971 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/d1bfccc6e46e4ad59529e90a933c0971 2023-04-23 05:54:53,101 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/63f3cd16999b460387a007ae5f7bf108 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/63f3cd16999b460387a007ae5f7bf108 2023-04-23 05:54:53,102 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/5cb0902dc7404ca3bb8db9ea29410ec4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/5cb0902dc7404ca3bb8db9ea29410ec4 2023-04-23 05:54:53,104 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5faf9ba22def4ee2a0a0a918e74c0350, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ff49df138ec34077a9c496389d1fc6cf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/cc56fedbc3e04b25884a5a4fcaae9cc4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/256e252a84bf4dbd833e3d2d88aa4302, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/e979bea0be0047bcaf8aa60b00a1fcde, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/2cd9ebb233534373837d12b03452f736, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d1a176a37cac4e84b9641aad65735004, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ec26691504ad4eaf9ea81348fa7398c2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/4858d24e32f74581812717d207dfbe66, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7924ae9cf2d74cf6b41444ea888f274b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ed82e7b138314f7e868154868f720ad4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/39c7a28c49604af2b974ca8026118c46, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3525f395ba3b48d5a565b49df909c29a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/f8552de6c94642ca8d2b579ac9ade5fa, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d26ae787cea641ac88942e0006c2f37c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5ef138dcfe614e899a594c80b89f7fc7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3b3a31f9e6b74ec7b91fc2a2bb85e2e5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/b7c3d648505d4e0898003d22fcc0d022, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/38902316882f4f6f8b132806e131b1ce, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/78bdcf66f9aa46f3b0ffced286c5a0c0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/731ae851000f4eee93e7f88b2ce94147, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/1f4d5c09461f49c0a17e528bcb69f12d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/43a3abf5e28b4085b79587d1139aa34a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/348b46a46df04894851ded2885165ca7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/76d9141ff5544756ba6a525bcba1a9c5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/c6d14693933f40f799aab1c7a6325034, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7de75b5c78834684a86a5a2c031fa5aa] to archive 2023-04-23 05:54:53,105 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:54:53,107 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5faf9ba22def4ee2a0a0a918e74c0350 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5faf9ba22def4ee2a0a0a918e74c0350 2023-04-23 05:54:53,108 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ff49df138ec34077a9c496389d1fc6cf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ff49df138ec34077a9c496389d1fc6cf 2023-04-23 05:54:53,109 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/cc56fedbc3e04b25884a5a4fcaae9cc4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/cc56fedbc3e04b25884a5a4fcaae9cc4 2023-04-23 05:54:53,110 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/256e252a84bf4dbd833e3d2d88aa4302 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/256e252a84bf4dbd833e3d2d88aa4302 2023-04-23 05:54:53,111 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/e979bea0be0047bcaf8aa60b00a1fcde to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/e979bea0be0047bcaf8aa60b00a1fcde 2023-04-23 05:54:53,112 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/2cd9ebb233534373837d12b03452f736 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/2cd9ebb233534373837d12b03452f736 2023-04-23 05:54:53,113 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d1a176a37cac4e84b9641aad65735004 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d1a176a37cac4e84b9641aad65735004 2023-04-23 05:54:53,114 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ec26691504ad4eaf9ea81348fa7398c2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ec26691504ad4eaf9ea81348fa7398c2 2023-04-23 05:54:53,115 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/4858d24e32f74581812717d207dfbe66 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/4858d24e32f74581812717d207dfbe66 2023-04-23 05:54:53,117 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7924ae9cf2d74cf6b41444ea888f274b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7924ae9cf2d74cf6b41444ea888f274b 2023-04-23 05:54:53,118 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ed82e7b138314f7e868154868f720ad4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/ed82e7b138314f7e868154868f720ad4 2023-04-23 05:54:53,119 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/39c7a28c49604af2b974ca8026118c46 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/39c7a28c49604af2b974ca8026118c46 2023-04-23 05:54:53,120 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3525f395ba3b48d5a565b49df909c29a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3525f395ba3b48d5a565b49df909c29a 2023-04-23 05:54:53,121 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/f8552de6c94642ca8d2b579ac9ade5fa to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/f8552de6c94642ca8d2b579ac9ade5fa 2023-04-23 05:54:53,122 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d26ae787cea641ac88942e0006c2f37c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/d26ae787cea641ac88942e0006c2f37c 2023-04-23 05:54:53,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5ef138dcfe614e899a594c80b89f7fc7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/5ef138dcfe614e899a594c80b89f7fc7 2023-04-23 05:54:53,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3b3a31f9e6b74ec7b91fc2a2bb85e2e5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/3b3a31f9e6b74ec7b91fc2a2bb85e2e5 2023-04-23 05:54:53,124 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/b7c3d648505d4e0898003d22fcc0d022 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/b7c3d648505d4e0898003d22fcc0d022 2023-04-23 05:54:53,125 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/38902316882f4f6f8b132806e131b1ce to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/38902316882f4f6f8b132806e131b1ce 2023-04-23 05:54:53,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/78bdcf66f9aa46f3b0ffced286c5a0c0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/78bdcf66f9aa46f3b0ffced286c5a0c0 2023-04-23 05:54:53,127 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/731ae851000f4eee93e7f88b2ce94147 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/731ae851000f4eee93e7f88b2ce94147 2023-04-23 05:54:53,128 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/1f4d5c09461f49c0a17e528bcb69f12d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/1f4d5c09461f49c0a17e528bcb69f12d 2023-04-23 05:54:53,129 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/43a3abf5e28b4085b79587d1139aa34a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/43a3abf5e28b4085b79587d1139aa34a 2023-04-23 05:54:53,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/348b46a46df04894851ded2885165ca7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/348b46a46df04894851ded2885165ca7 2023-04-23 05:54:53,131 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/76d9141ff5544756ba6a525bcba1a9c5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/76d9141ff5544756ba6a525bcba1a9c5 2023-04-23 05:54:53,132 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/c6d14693933f40f799aab1c7a6325034 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/c6d14693933f40f799aab1c7a6325034 2023-04-23 05:54:53,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7de75b5c78834684a86a5a2c031fa5aa to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/7de75b5c78834684a86a5a2c031fa5aa 2023-04-23 05:54:53,135 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d5942f77ea5f4ea1b2deb685e55e045f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/98c9eb649dad47e98f5db6190f4149f9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/903100e14b83414d9546a4bcda336a55, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/dd5bc9618f874fa195dfea11f34c1c22, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/745d70a5ee02400abde37447c50d2440, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f8f3fc92e1384ed6add0f5ac9ffc9654, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/9fd17d15e46f470ea8183cadc59130c1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/bc8cb2d546794b8188aabf949d7ea9bb, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/04a87b1145ee42dbbf25cd1d70b8e91b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/894557bf164b456f931225038d21cc34, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/acdbb257c04f4d21a063be3bc0262cba, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/03b6997d60b14ad58a85a3f5a257c9b9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/a6ec29aa2c314a79838e2f72474ec403, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/435a967589e540108e0fb7a0fd8b5d2a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f06c8289543b4797a4e1febeeed18c18, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/c0bb7dd20a1247e9896104f52537ca27, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/1f2fe727f8e84762a05b34cd94a76edf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ed8c2f0041f34c02b1cdd112eb25da71, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/26279c63d5eb44e28156bb3e481fc73f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/97e6a2c253ef445392b3cd1191058c2d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/4dc2382b83224a5bacd18bae2ac455b7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/2ec66d3ff677459f8fd0d04e38563980, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ca0a09164f8443dc95d7bc9851080127, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d7dd17d0a6e94f3692300dc27f466b52, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/023321e4c3e24997b5e2d9324e0499c5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/0b84a9982da6454c8009b50934b9d87a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/be464d4419794cc88c84a83227aca552] to archive 2023-04-23 05:54:53,136 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:54:53,137 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d5942f77ea5f4ea1b2deb685e55e045f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d5942f77ea5f4ea1b2deb685e55e045f 2023-04-23 05:54:53,138 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/98c9eb649dad47e98f5db6190f4149f9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/98c9eb649dad47e98f5db6190f4149f9 2023-04-23 05:54:53,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/903100e14b83414d9546a4bcda336a55 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/903100e14b83414d9546a4bcda336a55 2023-04-23 05:54:53,140 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/dd5bc9618f874fa195dfea11f34c1c22 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/dd5bc9618f874fa195dfea11f34c1c22 2023-04-23 05:54:53,141 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/745d70a5ee02400abde37447c50d2440 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/745d70a5ee02400abde37447c50d2440 2023-04-23 05:54:53,142 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f8f3fc92e1384ed6add0f5ac9ffc9654 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f8f3fc92e1384ed6add0f5ac9ffc9654 2023-04-23 05:54:53,143 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/9fd17d15e46f470ea8183cadc59130c1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/9fd17d15e46f470ea8183cadc59130c1 2023-04-23 05:54:53,143 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/bc8cb2d546794b8188aabf949d7ea9bb to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/bc8cb2d546794b8188aabf949d7ea9bb 2023-04-23 05:54:53,144 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/04a87b1145ee42dbbf25cd1d70b8e91b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/04a87b1145ee42dbbf25cd1d70b8e91b 2023-04-23 05:54:53,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/894557bf164b456f931225038d21cc34 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/894557bf164b456f931225038d21cc34 2023-04-23 05:54:53,146 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/acdbb257c04f4d21a063be3bc0262cba to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/acdbb257c04f4d21a063be3bc0262cba 2023-04-23 05:54:53,147 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/03b6997d60b14ad58a85a3f5a257c9b9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/03b6997d60b14ad58a85a3f5a257c9b9 2023-04-23 05:54:53,148 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/a6ec29aa2c314a79838e2f72474ec403 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/a6ec29aa2c314a79838e2f72474ec403 2023-04-23 05:54:53,149 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/435a967589e540108e0fb7a0fd8b5d2a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/435a967589e540108e0fb7a0fd8b5d2a 2023-04-23 05:54:53,149 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f06c8289543b4797a4e1febeeed18c18 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/f06c8289543b4797a4e1febeeed18c18 2023-04-23 05:54:53,150 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/c0bb7dd20a1247e9896104f52537ca27 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/c0bb7dd20a1247e9896104f52537ca27 2023-04-23 05:54:53,151 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/1f2fe727f8e84762a05b34cd94a76edf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/1f2fe727f8e84762a05b34cd94a76edf 2023-04-23 05:54:53,152 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ed8c2f0041f34c02b1cdd112eb25da71 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ed8c2f0041f34c02b1cdd112eb25da71 2023-04-23 05:54:53,153 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/26279c63d5eb44e28156bb3e481fc73f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/26279c63d5eb44e28156bb3e481fc73f 2023-04-23 05:54:53,154 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/97e6a2c253ef445392b3cd1191058c2d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/97e6a2c253ef445392b3cd1191058c2d 2023-04-23 05:54:53,156 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/4dc2382b83224a5bacd18bae2ac455b7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/4dc2382b83224a5bacd18bae2ac455b7 2023-04-23 05:54:53,157 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/2ec66d3ff677459f8fd0d04e38563980 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/2ec66d3ff677459f8fd0d04e38563980 2023-04-23 05:54:53,158 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ca0a09164f8443dc95d7bc9851080127 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/ca0a09164f8443dc95d7bc9851080127 2023-04-23 05:54:53,159 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d7dd17d0a6e94f3692300dc27f466b52 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/d7dd17d0a6e94f3692300dc27f466b52 2023-04-23 05:54:53,160 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/023321e4c3e24997b5e2d9324e0499c5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/023321e4c3e24997b5e2d9324e0499c5 2023-04-23 05:54:53,161 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/0b84a9982da6454c8009b50934b9d87a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/0b84a9982da6454c8009b50934b9d87a 2023-04-23 05:54:53,162 DEBUG [StoreCloser-TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/be464d4419794cc88c84a83227aca552 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/be464d4419794cc88c84a83227aca552 2023-04-23 05:54:53,167 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/recovered.edits/479.seqid, newMaxSeqId=479, maxSeqId=4 2023-04-23 05:54:53,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717. 2023-04-23 05:54:53,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ba2bf059880117cc12577e45cc6d7717: 2023-04-23 05:54:53,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:53,170 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=ba2bf059880117cc12577e45cc6d7717, regionState=CLOSED 2023-04-23 05:54:53,170 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229293170"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229293170"}]},"ts":"1682229293170"} 2023-04-23 05:54:53,172 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-23 05:54:53,173 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure ba2bf059880117cc12577e45cc6d7717, server=jenkins-hbase4.apache.org,40995,1682229223026 in 1.9280 sec 2023-04-23 05:54:53,174 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-23 05:54:53,174 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ba2bf059880117cc12577e45cc6d7717, UNASSIGN in 1.9330 sec 2023-04-23 05:54:53,175 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229293174"}]},"ts":"1682229293174"} 2023-04-23 05:54:53,175 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-23 05:54:53,177 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-23 05:54:53,178 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9610 sec 2023-04-23 05:54:53,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-23 05:54:53,328 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-23 05:54:53,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-23 05:54:53,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:53,331 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:53,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-23 05:54:53,332 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:53,337 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:53,339 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/recovered.edits] 2023-04-23 05:54:53,342 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/240ff99259a6418b8029e63bb80a3bde to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/240ff99259a6418b8029e63bb80a3bde 2023-04-23 05:54:53,343 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c7059ac676164e8a94359869405ee158 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/A/c7059ac676164e8a94359869405ee158 2023-04-23 05:54:53,345 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/065257578fbb4933be5bb7cfd001cc9d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/065257578fbb4933be5bb7cfd001cc9d 2023-04-23 05:54:53,346 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/0ebe85af7485471891c6d4eb44d4a654 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/B/0ebe85af7485471891c6d4eb44d4a654 2023-04-23 05:54:53,349 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/45af3f4b905d4cfd966634af71879849 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/45af3f4b905d4cfd966634af71879849 2023-04-23 05:54:53,350 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/754c5e966bc442ffae4897d828493f66 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/C/754c5e966bc442ffae4897d828493f66 2023-04-23 05:54:53,353 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/recovered.edits/479.seqid to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717/recovered.edits/479.seqid 2023-04-23 05:54:53,353 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/ba2bf059880117cc12577e45cc6d7717 2023-04-23 05:54:53,353 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:54:53,354 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-23 05:54:53,354 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-23 05:54:53,358 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304230a220c7bb3b943f48de5113b6f0707f7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304230a220c7bb3b943f48de5113b6f0707f7 2023-04-23 05:54:53,359 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042311cd52a9fb3d44d69ad2fe103fa271e3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042311cd52a9fb3d44d69ad2fe103fa271e3 2023-04-23 05:54:53,360 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042319a897453e5b42c28caa669481491880 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042319a897453e5b42c28caa669481491880 2023-04-23 05:54:53,361 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233153c0b33a764b9f9324e6f8ec5ce837 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233153c0b33a764b9f9324e6f8ec5ce837 2023-04-23 05:54:53,362 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423327eb7f912314dd39a4fdae2d0ad817a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423327eb7f912314dd39a4fdae2d0ad817a 2023-04-23 05:54:53,363 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233af71a45465046bd980d6ec7a7cdb295 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233af71a45465046bd980d6ec7a7cdb295 2023-04-23 05:54:53,364 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423472e0428af944094baf676fdaddff8f1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423472e0428af944094baf676fdaddff8f1 2023-04-23 05:54:53,365 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042347b891a6ab7a4e8f962f6e9074c93361 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042347b891a6ab7a4e8f962f6e9074c93361 2023-04-23 05:54:53,366 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236562799b1bc447f9abe067ac2fde2bbd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236562799b1bc447f9abe067ac2fde2bbd 2023-04-23 05:54:53,368 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042366275d30489240dea063c030b3f8321c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042366275d30489240dea063c030b3f8321c 2023-04-23 05:54:53,369 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236b51ed1f98f0487486e316d51e13a82d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236b51ed1f98f0487486e316d51e13a82d 2023-04-23 05:54:53,370 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304237a775a9fa1fd42f19a5e609c66940bc6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304237a775a9fa1fd42f19a5e609c66940bc6 2023-04-23 05:54:53,371 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423847c6749fa314f97a75cbe8fffcd4442 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423847c6749fa314f97a75cbe8fffcd4442 2023-04-23 05:54:53,373 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304238e0b5d0fe9604a75a0a84e55a77dccb1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304238e0b5d0fe9604a75a0a84e55a77dccb1 2023-04-23 05:54:53,374 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304238e6fd3e0aaf24372b18be78c2a2b55e3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304238e6fd3e0aaf24372b18be78c2a2b55e3 2023-04-23 05:54:53,375 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a1e4b4df4e7f4b2caf7585f8236467e3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a1e4b4df4e7f4b2caf7585f8236467e3 2023-04-23 05:54:53,376 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a5d3a146f72e459ba0743302e5497d1a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a5d3a146f72e459ba0743302e5497d1a 2023-04-23 05:54:53,378 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c3f9c8475fbc45288f17cfdbc6479bc3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c3f9c8475fbc45288f17cfdbc6479bc3 2023-04-23 05:54:53,379 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d4a024855253479a9603a2b8bac9dac3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d4a024855253479a9603a2b8bac9dac3 2023-04-23 05:54:53,380 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d4dbe4af70784bbb97fec3ce9c37b20b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d4dbe4af70784bbb97fec3ce9c37b20b 2023-04-23 05:54:53,381 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e01d8c94343745a6a378b9fd109e4100 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e01d8c94343745a6a378b9fd109e4100 2023-04-23 05:54:53,382 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423ebcb920453b84889bd366141566add18 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423ebcb920453b84889bd366141566add18 2023-04-23 05:54:53,383 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f55e92d169dd4f37bde06da40fca5b4e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f55e92d169dd4f37bde06da40fca5b4e 2023-04-23 05:54:53,384 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-23 05:54:53,386 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:53,388 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-23 05:54:53,391 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-23 05:54:53,392 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:53,392 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-23 05:54:53,392 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682229293392"}]},"ts":"9223372036854775807"} 2023-04-23 05:54:53,394 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-23 05:54:53,394 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ba2bf059880117cc12577e45cc6d7717, NAME => 'TestAcidGuarantees,,1682229261413.ba2bf059880117cc12577e45cc6d7717.', STARTKEY => '', ENDKEY => ''}] 2023-04-23 05:54:53,394 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-23 05:54:53,394 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682229293394"}]},"ts":"9223372036854775807"} 2023-04-23 05:54:53,396 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-23 05:54:53,398 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:53,399 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 69 msec 2023-04-23 05:54:53,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-23 05:54:53,433 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-23 05:54:53,446 INFO [Listener at localhost/41845] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 232), OpenFileDescriptor=541 (was 530) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=366 (was 405), ProcessCount=169 (was 170), AvailableMemoryMB=12381 (was 12400) 2023-04-23 05:54:53,458 INFO [Listener at localhost/41845] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=541, MaxFileDescriptor=60000, SystemLoadAverage=366, ProcessCount=169, AvailableMemoryMB=12381 2023-04-23 05:54:53,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-23 05:54:53,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:54:53,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-23 05:54:53,463 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-23 05:54:53,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-23 05:54:53,464 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-23 05:54:53,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-23 05:54:53,465 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:53,465 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd empty. 2023-04-23 05:54:53,466 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:53,466 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:54:53,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-23 05:54:53,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-23 05:54:53,874 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-23 05:54:53,876 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 911c079a878d9ede8370cba6d74e3efd, NAME => 'TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp 2023-04-23 05:54:54,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-23 05:54:54,282 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:54:54,282 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 911c079a878d9ede8370cba6d74e3efd, disabling compactions & flushes 2023-04-23 05:54:54,282 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:54,282 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:54,282 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. after waiting 0 ms 2023-04-23 05:54:54,282 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:54,282 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:54,282 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:54,284 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-23 05:54:54,285 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229294285"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229294285"}]},"ts":"1682229294285"} 2023-04-23 05:54:54,287 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-23 05:54:54,287 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-23 05:54:54,287 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229294287"}]},"ts":"1682229294287"} 2023-04-23 05:54:54,288 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-23 05:54:54,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=911c079a878d9ede8370cba6d74e3efd, ASSIGN}] 2023-04-23 05:54:54,292 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=911c079a878d9ede8370cba6d74e3efd, ASSIGN 2023-04-23 05:54:54,293 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=911c079a878d9ede8370cba6d74e3efd, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=false 2023-04-23 05:54:54,444 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=911c079a878d9ede8370cba6d74e3efd, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:54,444 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229294444"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229294444"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229294444"}]},"ts":"1682229294444"} 2023-04-23 05:54:54,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:54:54,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-23 05:54:54,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:54,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 911c079a878d9ede8370cba6d74e3efd, NAME => 'TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:54:54,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:54,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:54:54,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:54,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:54,602 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:54,604 DEBUG [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A 2023-04-23 05:54:54,604 DEBUG [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A 2023-04-23 05:54:54,604 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:54:54,604 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 911c079a878d9ede8370cba6d74e3efd columnFamilyName A 2023-04-23 05:54:54,605 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] regionserver.HStore(310): Store=911c079a878d9ede8370cba6d74e3efd/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:54:54,605 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:54,606 DEBUG [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B 2023-04-23 05:54:54,606 DEBUG [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B 2023-04-23 05:54:54,607 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:54:54,607 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 911c079a878d9ede8370cba6d74e3efd columnFamilyName B 2023-04-23 05:54:54,608 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] regionserver.HStore(310): Store=911c079a878d9ede8370cba6d74e3efd/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:54:54,608 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:54,609 DEBUG [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C 2023-04-23 05:54:54,609 DEBUG [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C 2023-04-23 05:54:54,609 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:54:54,609 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 911c079a878d9ede8370cba6d74e3efd columnFamilyName C 2023-04-23 05:54:54,610 INFO [StoreOpener-911c079a878d9ede8370cba6d74e3efd-1] regionserver.HStore(310): Store=911c079a878d9ede8370cba6d74e3efd/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:54:54,610 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:54,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:54,611 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:54,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:54:54,614 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:54,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-23 05:54:54,616 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 911c079a878d9ede8370cba6d74e3efd; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65009207, jitterRate=-0.03128732740879059}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:54:54,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:54,617 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., pid=30, masterSystemTime=1682229294597 2023-04-23 05:54:54,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:54,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:54,619 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=911c079a878d9ede8370cba6d74e3efd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:54,619 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229294619"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229294619"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229294619"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229294619"}]},"ts":"1682229294619"} 2023-04-23 05:54:54,622 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-23 05:54:54,622 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 in 174 msec 2023-04-23 05:54:54,623 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-23 05:54:54,623 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=911c079a878d9ede8370cba6d74e3efd, ASSIGN in 331 msec 2023-04-23 05:54:54,624 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-23 05:54:54,624 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229294624"}]},"ts":"1682229294624"} 2023-04-23 05:54:54,625 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-23 05:54:54,627 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-23 05:54:54,628 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1660 sec 2023-04-23 05:54:55,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-23 05:54:55,569 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-23 05:54:55,570 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x53921272 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,576 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d7cdc2b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,577 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,578 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52784, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,579 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-23 05:54:55,581 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56286, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-23 05:54:55,582 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x13544701 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,587 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5096c69b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,587 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x10b79c2c to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,591 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@216aad74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,591 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5a81fa to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,594 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76203061, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,595 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x3562c7bf to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,599 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b39d5b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,599 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x6705d115 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,602 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44f9bd75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,603 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x5e5b1c14 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,606 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@574000bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,606 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x4f29c1c4 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,611 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4994df89, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,611 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x681ace6c to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,616 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69bc154a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,617 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x2b4fa5d3 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,621 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15186778, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,621 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x45de5ad8 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:54:55,625 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13e90e8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:54:55,629 DEBUG [hconnection-0x5f9f91f7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,629 DEBUG [hconnection-0x324f476c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,629 DEBUG [hconnection-0x54489ca2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,630 DEBUG [hconnection-0x43d154a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,630 DEBUG [hconnection-0x61eb9d5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,630 DEBUG [hconnection-0x3469fe8f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,630 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52794, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,631 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52816, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,637 DEBUG [hconnection-0x42bb4f50-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,638 DEBUG [hconnection-0x5ae59228-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,637 DEBUG [hconnection-0x492b5c1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,640 DEBUG [hconnection-0x1b3465c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:54:55,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:55,641 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52830, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,644 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52788, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,645 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52810, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,645 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52826, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,645 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,646 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52804, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,646 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52824, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,646 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52844, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:54:55,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:55,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:55,654 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:55,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:55,655 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:55,655 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:55,657 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,658 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:55,659 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:55,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:55,659 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,659 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:55,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:55,666 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:54:55,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:54:55,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:55,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:54:55,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:55,666 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:54:55,666 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:55,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:55,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:55,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:55,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:55,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:55,678 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:55,678 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:55,678 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:55,678 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:55,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:55,682 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:55,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6830bc44[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:55,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:55,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:55,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:55,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:55,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:55,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:55,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:55,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:55,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:55,691 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:55,691 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:55,691 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:55,691 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:55,691 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:55,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:55,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:55,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:55,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229355688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:55,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229355690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229295686, max timestamp=1682229295692Num uniques 859; 2023-04-23 05:54:55,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:55,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229295686, max timestamp=1682229295692Num uniques 859; 2023-04-23 05:54:55,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:55,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229295686, max timestamp=1682229295692Num uniques 859; 2023-04-23 05:54:55,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:55,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:55,696 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:55,705 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:55,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/952f5966f5e8401582e1233a7fddf233 2023-04-23 05:54:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:55,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:55,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:55,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:55,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:55,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:55,715 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:55,715 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:55,715 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:55,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:55,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:55,715 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:55,715 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:55,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:55,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52826 deadline: 1682229355715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:55,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:55,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229355719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52830 deadline: 1682229355719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/fa923bced2a94633bc343c1a71557cd5 2023-04-23 05:54:55,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/87498d8d9df44d829839b75ab9cb514a 2023-04-23 05:54:55,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/952f5966f5e8401582e1233a7fddf233 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/952f5966f5e8401582e1233a7fddf233 2023-04-23 05:54:55,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:55,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229355796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:55,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229355796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/952f5966f5e8401582e1233a7fddf233, entries=150, sequenceid=12, filesize=11.6 K 2023-04-23 05:54:55,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/fa923bced2a94633bc343c1a71557cd5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fa923bced2a94633bc343c1a71557cd5 2023-04-23 05:54:55,808 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fa923bced2a94633bc343c1a71557cd5, entries=150, sequenceid=12, filesize=11.6 K 2023-04-23 05:54:55,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/87498d8d9df44d829839b75ab9cb514a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/87498d8d9df44d829839b75ab9cb514a 2023-04-23 05:54:55,815 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:55,817 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/87498d8d9df44d829839b75ab9cb514a, entries=150, sequenceid=12, filesize=11.6 K 2023-04-23 05:54:55,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:55,819 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 911c079a878d9ede8370cba6d74e3efd in 153ms, sequenceid=12, compaction requested=false 2023-04-23 05:54:55,819 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:55,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:55,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:55,827 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:55,827 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:55,827 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:55,827 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:55,827 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,828 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:55,829 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:55,829 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:55,829 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:55,829 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:55,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:55,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:55,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:55,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:55,832 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:55,832 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:55,832 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-23 05:54:55,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:55,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:55,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9968, totalHeapSize=22912, min timestamp=1682229295712, max timestamp=1682229295829Num uniques 1018; 2023-04-23 05:54:55,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46608, new segment data size=9968, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-23 05:54:55,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9968, totalHeapSize=22912, min timestamp=1682229295712, max timestamp=1682229295829Num uniques 1018; 2023-04-23 05:54:55,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46608, new segment data size=9968, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-23 05:54:55,871 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:54:55,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:55,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:54:55,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:55,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:54:55,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1682229295712, max timestamp=1682229295829Num uniques 1018; 2023-04-23 05:54:55,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46608, new segment data size=9058, suffix heap size=133840,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=198 2023-04-23 05:54:55,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:55,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229295894, max timestamp=1682229295927Num uniques 859; 2023-04-23 05:54:55,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229295894, max timestamp=1682229295927Num uniques 859; 2023-04-23 05:54:55,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229295894, max timestamp=1682229295927Num uniques 859; 2023-04-23 05:54:55,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:55,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:55,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:55,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229295943, max timestamp=1682229295949Num uniques 1018; 2023-04-23 05:54:55,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229295943, max timestamp=1682229295949Num uniques 1018; 2023-04-23 05:54:55,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:54:55,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:54:55,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229295943, max timestamp=1682229295949Num uniques 1018; 2023-04-23 05:54:55,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:54:55,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229295983, max timestamp=1682229295994Num uniques 1027; 2023-04-23 05:54:55,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229295983, max timestamp=1682229295994Num uniques 1027; 2023-04-23 05:54:55,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-23 05:54:55,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229295983, max timestamp=1682229295994Num uniques 1027; 2023-04-23 05:54:55,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-23 05:54:55,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-23 05:54:56,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229356009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229356009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8507, totalHeapSize=19584, min timestamp=1682229296008, max timestamp=1682229296011Num uniques 1036; 2023-04-23 05:54:56,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47437, new segment data size=8507, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-23 05:54:56,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229356010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1682229296008, max timestamp=1682229296011Num uniques 1036; 2023-04-23 05:54:56,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1682229296008, max timestamp=1682229296011Num uniques 1036; 2023-04-23 05:54:56,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=8506, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-23 05:54:56,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=8506, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-23 05:54:56,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682229296023, max timestamp=1682229296034Num uniques 1045; 2023-04-23 05:54:56,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-23 05:54:56,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682229296023, max timestamp=1682229296034Num uniques 1045; 2023-04-23 05:54:56,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-23 05:54:56,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682229296023, max timestamp=1682229296034Num uniques 1045; 2023-04-23 05:54:56,037 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-23 05:54:56,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52830 deadline: 1682229356090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52826 deadline: 1682229356090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229356113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229356114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229356114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52830 deadline: 1682229356195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52826 deadline: 1682229356195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,294 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/416217d6429141af9cc12dd224e31e2b 2023-04-23 05:54:56,307 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/ec2c02fc590c4408a17654639f93a413 2023-04-23 05:54:56,317 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=12.53 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/acb3991d58004197b238a56ec676c568 2023-04-23 05:54:56,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229356315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229356317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229356318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,321 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/416217d6429141af9cc12dd224e31e2b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/416217d6429141af9cc12dd224e31e2b 2023-04-23 05:54:56,325 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/416217d6429141af9cc12dd224e31e2b, entries=150, sequenceid=51, filesize=11.6 K 2023-04-23 05:54:56,326 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/ec2c02fc590c4408a17654639f93a413 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec2c02fc590c4408a17654639f93a413 2023-04-23 05:54:56,330 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec2c02fc590c4408a17654639f93a413, entries=150, sequenceid=51, filesize=11.6 K 2023-04-23 05:54:56,330 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/acb3991d58004197b238a56ec676c568 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/acb3991d58004197b238a56ec676c568 2023-04-23 05:54:56,334 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/acb3991d58004197b238a56ec676c568, entries=150, sequenceid=51, filesize=11.6 K 2023-04-23 05:54:56,335 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~39.37 KB/40310, heapSize ~98.47 KB/100832, currentSize=147.60 KB/151140 for 911c079a878d9ede8370cba6d74e3efd in 503ms, sequenceid=51, compaction requested=false 2023-04-23 05:54:56,335 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:56,335 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:56,335 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:56,336 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,336 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:56,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cd1fe37[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:56,338 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:56,339 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,340 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:56,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:56,340 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,340 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:56,342 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:56,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:56,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:56,345 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:56,346 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:56,346 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:56,346 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:56,346 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:56,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,348 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,348 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,352 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,352 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:56,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:56,352 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,352 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,352 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,352 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,352 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,352 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,352 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:56,352 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,352 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:56,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:56,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:56,399 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=154.31 KB heapSize=437.39 KB 2023-04-23 05:54:56,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:54:56,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-23 05:54:56,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:54:56,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-23 05:54:56,400 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:54:56,400 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-23 05:54:56,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52830 deadline: 1682229356407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52826 deadline: 1682229356408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,453 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:56,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:56,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:56,456 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:56,456 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,456 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:56,456 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:56,457 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,457 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,458 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,458 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:56,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:56,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:56,460 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:56,460 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,460 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:56,460 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:56,460 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,461 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:56,461 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:56,461 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,461 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:56,461 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:56,463 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:56,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@435a81f7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:56,465 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:56,465 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,469 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,469 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:56,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:56,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:56,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:56,471 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,471 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:56,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:56,474 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:56,474 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:56,474 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:56,474 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:56,474 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:56,478 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,478 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,478 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,479 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,480 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:56,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:56,483 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:56,483 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:56,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:56,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52830 deadline: 1682229356510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52826 deadline: 1682229356512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,584 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:56,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:56,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:56,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:56,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:56,589 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:56,589 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,589 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:56,589 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:56,589 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,590 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,591 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,591 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,591 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,591 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:56,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:56,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:56,593 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,593 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:56,593 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:56,593 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:56,593 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,595 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bfee81f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:56,597 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:56,597 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,599 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,599 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:56,599 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:56,599 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,599 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:56,599 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:56,601 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,601 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:56,603 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:56,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:56,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:56,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:56,604 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:56,605 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,605 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,605 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,606 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,610 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:56,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:56,611 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:56,611 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:56,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229356619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229356620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229356622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,711 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:56,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:56,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:56,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:56,714 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:56,714 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,714 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:56,714 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:56,715 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,715 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52830 deadline: 1682229356713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:56,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52826 deadline: 1682229356714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,721 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,721 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,721 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,721 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:56,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:56,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:56,723 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,723 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:56,723 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:56,723 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:56,723 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:56,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,724 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,725 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,725 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:56,726 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:56,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@641abb8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:56,726 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,727 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,727 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,727 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:56,727 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:56,727 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:56,727 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:56,729 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,729 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,729 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,729 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:56,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:56,731 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:56,731 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:56,731 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:56,731 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:56,731 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:56,734 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,734 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,734 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,735 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,736 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:56,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:56,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,739 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:56,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,739 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:56,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/8da22d8a626d409c831d20d4d8fd01f9 2023-04-23 05:54:56,840 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:56,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:56,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:56,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:56,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:56,845 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:56,845 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,845 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:56,845 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:56,845 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,845 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:56,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:56,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:56,849 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,849 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:56,849 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:56,849 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,851 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:56,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6582cd01[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:56,853 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:56,853 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/0df1c0a6b9ad4c16ada7b6953356aeab 2023-04-23 05:54:56,856 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:56,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:56,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:56,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:56,860 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:56,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:56,863 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:56,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:56,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:56,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:56,863 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:56,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,867 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:56,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:56,867 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/ab99a66694a14def97117c3d1f1c3e27 2023-04-23 05:54:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:56,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:56,872 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/8da22d8a626d409c831d20d4d8fd01f9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/8da22d8a626d409c831d20d4d8fd01f9 2023-04-23 05:54:56,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:56,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:56,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:56,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:56,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:56,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:56,875 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:56,875 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:56,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:56,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/8da22d8a626d409c831d20d4d8fd01f9, entries=150, sequenceid=158, filesize=11.8 K 2023-04-23 05:54:56,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/0df1c0a6b9ad4c16ada7b6953356aeab as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/0df1c0a6b9ad4c16ada7b6953356aeab 2023-04-23 05:54:56,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/0df1c0a6b9ad4c16ada7b6953356aeab, entries=150, sequenceid=158, filesize=11.8 K 2023-04-23 05:54:56,882 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/ab99a66694a14def97117c3d1f1c3e27 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ab99a66694a14def97117c3d1f1c3e27 2023-04-23 05:54:56,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ab99a66694a14def97117c3d1f1c3e27, entries=150, sequenceid=158, filesize=11.8 K 2023-04-23 05:54:56,887 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~437.34 KB/447840, currentSize=26.84 KB/27480 for 911c079a878d9ede8370cba6d74e3efd in 488ms, sequenceid=158, compaction requested=true 2023-04-23 05:54:56,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:56,887 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:56,887 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:56,887 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:56,887 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:56,887 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:56,888 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35808 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:56,888 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35808 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:56,888 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/A is initiating minor compaction (all files) 2023-04-23 05:54:56,888 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/B is initiating minor compaction (all files) 2023-04-23 05:54:56,888 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/A in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,888 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/B in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,888 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/952f5966f5e8401582e1233a7fddf233, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/416217d6429141af9cc12dd224e31e2b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/8da22d8a626d409c831d20d4d8fd01f9] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=35.0 K 2023-04-23 05:54:56,888 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fa923bced2a94633bc343c1a71557cd5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec2c02fc590c4408a17654639f93a413, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/0df1c0a6b9ad4c16ada7b6953356aeab] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=35.0 K 2023-04-23 05:54:56,889 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 952f5966f5e8401582e1233a7fddf233, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682229295648 2023-04-23 05:54:56,889 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting fa923bced2a94633bc343c1a71557cd5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682229295648 2023-04-23 05:54:56,889 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 416217d6429141af9cc12dd224e31e2b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682229295717 2023-04-23 05:54:56,889 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ec2c02fc590c4408a17654639f93a413, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682229295717 2023-04-23 05:54:56,892 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 8da22d8a626d409c831d20d4d8fd01f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682229296073 2023-04-23 05:54:56,892 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 0df1c0a6b9ad4c16ada7b6953356aeab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682229296073 2023-04-23 05:54:56,901 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#A#compaction#201 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:54:56,901 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#B#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:54:56,935 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/e515e9c08b2a432f925462896f08f187 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/e515e9c08b2a432f925462896f08f187 2023-04-23 05:54:56,940 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/B of 911c079a878d9ede8370cba6d74e3efd into e515e9c08b2a432f925462896f08f187(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:56,940 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:56,940 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/B, priority=13, startTime=1682229296887; duration=0sec 2023-04-23 05:54:56,941 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:56,941 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:56,942 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35808 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:56,942 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/C is initiating minor compaction (all files) 2023-04-23 05:54:56,942 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/C in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,942 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/87498d8d9df44d829839b75ab9cb514a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/acb3991d58004197b238a56ec676c568, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ab99a66694a14def97117c3d1f1c3e27] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=35.0 K 2023-04-23 05:54:56,942 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 87498d8d9df44d829839b75ab9cb514a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1682229295648 2023-04-23 05:54:56,942 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting acb3991d58004197b238a56ec676c568, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682229295717 2023-04-23 05:54:56,943 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ab99a66694a14def97117c3d1f1c3e27, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682229296073 2023-04-23 05:54:56,948 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#C#compaction#202 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:54:56,967 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/031f33cc2ec0496cbfcf7ef43421ff4c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/031f33cc2ec0496cbfcf7ef43421ff4c 2023-04-23 05:54:56,971 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/C of 911c079a878d9ede8370cba6d74e3efd into 031f33cc2ec0496cbfcf7ef43421ff4c(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:56,971 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:56,971 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/C, priority=13, startTime=1682229296887; duration=0sec 2023-04-23 05:54:56,971 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:56,975 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:56,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:56,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:56,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:56,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:56,979 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:56,979 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:56,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:56,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,981 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:56,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:56,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:56,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:56,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:56,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:56,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:56,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:56,983 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:56,983 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:56,983 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-23 05:54:56,983 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:54:56,983 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:56,983 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:54:56,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:56,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:54:56,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:57,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:57,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:57,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229297038, max timestamp=1682229297042Num uniques 859; 2023-04-23 05:54:57,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229297038, max timestamp=1682229297042Num uniques 859; 2023-04-23 05:54:57,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:57,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:57,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229297038, max timestamp=1682229297042Num uniques 859; 2023-04-23 05:54:57,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:54:57,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229297055, max timestamp=1682229297072Num uniques 1018; 2023-04-23 05:54:57,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:54:57,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229297055, max timestamp=1682229297072Num uniques 1018; 2023-04-23 05:54:57,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:54:57,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229297055, max timestamp=1682229297072Num uniques 1018; 2023-04-23 05:54:57,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:54:57,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682229297086, max timestamp=1682229297103Num uniques 1027; 2023-04-23 05:54:57,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682229297086, max timestamp=1682229297103Num uniques 1027; 2023-04-23 05:54:57,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682229297086, max timestamp=1682229297103Num uniques 1027; 2023-04-23 05:54:57,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-23 05:54:57,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-23 05:54:57,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-23 05:54:57,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682229297125, max timestamp=1682229297131Num uniques 1086; 2023-04-23 05:54:57,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-23 05:54:57,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682229297125, max timestamp=1682229297131Num uniques 1086; 2023-04-23 05:54:57,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-23 05:54:57,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682229297125, max timestamp=1682229297131Num uniques 1086; 2023-04-23 05:54:57,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-23 05:54:57,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682229297142, max timestamp=1682229297147Num uniques 1145; 2023-04-23 05:54:57,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-23 05:54:57,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682229297142, max timestamp=1682229297147Num uniques 1145; 2023-04-23 05:54:57,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-23 05:54:57,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682229297142, max timestamp=1682229297147Num uniques 1145; 2023-04-23 05:54:57,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-23 05:54:57,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682229297182, max timestamp=1682229297188Num uniques 1104; 2023-04-23 05:54:57,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-23 05:54:57,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682229297182, max timestamp=1682229297188Num uniques 1104; 2023-04-23 05:54:57,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682229297182, max timestamp=1682229297188Num uniques 1104; 2023-04-23 05:54:57,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-23 05:54:57,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-23 05:54:57,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52830 deadline: 1682229357226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682229297194, max timestamp=1682229297227Num uniques 1063; 2023-04-23 05:54:57,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682229297194, max timestamp=1682229297227Num uniques 1063; 2023-04-23 05:54:57,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=14328, suffix heap size=138520,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=313 2023-04-23 05:54:57,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=14328, suffix heap size=138520,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=313 2023-04-23 05:54:57,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1682229297194, max timestamp=1682229297227Num uniques 1063; 2023-04-23 05:54:57,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=14328, suffix heap size=138520,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=313 2023-04-23 05:54:57,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229357241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52826 deadline: 1682229357243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229357243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682229297237, max timestamp=1682229297243Num uniques 1172; 2023-04-23 05:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-04-23 05:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682229297237, max timestamp=1682229297243Num uniques 1172; 2023-04-23 05:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1682229297237, max timestamp=1682229297243Num uniques 1172; 2023-04-23 05:54:57,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-04-23 05:54:57,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-04-23 05:54:57,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682229297279, max timestamp=1682229297301Num uniques 1081; 2023-04-23 05:54:57,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-23 05:54:57,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682229297279, max timestamp=1682229297301Num uniques 1081; 2023-04-23 05:54:57,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682229297279, max timestamp=1682229297301Num uniques 1081; 2023-04-23 05:54:57,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-23 05:54:57,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-23 05:54:57,337 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/e0d6d4797cb8452bba1215755d0b1766 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/e0d6d4797cb8452bba1215755d0b1766 2023-04-23 05:54:57,342 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/A of 911c079a878d9ede8370cba6d74e3efd into e0d6d4797cb8452bba1215755d0b1766(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:57,342 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:57,342 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/A, priority=13, startTime=1682229296887; duration=0sec 2023-04-23 05:54:57,342 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:57,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682229297338, max timestamp=1682229297349Num uniques 1040; 2023-04-23 05:54:57,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682229297338, max timestamp=1682229297349Num uniques 1040; 2023-04-23 05:54:57,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682229297338, max timestamp=1682229297349Num uniques 1040; 2023-04-23 05:54:57,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-23 05:54:57,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-23 05:54:57,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-23 05:54:57,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682229297355, max timestamp=1682229297366Num uniques 1099; 2023-04-23 05:54:57,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-23 05:54:57,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682229297355, max timestamp=1682229297366Num uniques 1099; 2023-04-23 05:54:57,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-23 05:54:57,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1682229297355, max timestamp=1682229297366Num uniques 1099; 2023-04-23 05:54:57,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=13694, suffix heap size=142264,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=299 2023-04-23 05:54:57,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1682229297376, max timestamp=1682229297382Num uniques 1058; 2023-04-23 05:54:57,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1682229297376, max timestamp=1682229297382Num uniques 1158; 2023-04-23 05:54:57,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-23 05:54:57,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1682229297376, max timestamp=1682229297382Num uniques 1058; 2023-04-23 05:54:57,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53028, new segment data size=9518, suffix heap size=148400,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=208 2023-04-23 05:54:57,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-23 05:54:57,392 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/3059e55e5f48480693697a18c1ac9a43 2023-04-23 05:54:57,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229357398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229357400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9922, totalHeapSize=22808, min timestamp=1682229297395, max timestamp=1682229297402Num uniques 1067; 2023-04-23 05:54:57,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48852, new segment data size=9922, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-23 05:54:57,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9922, totalHeapSize=22808, min timestamp=1682229297395, max timestamp=1682229297402Num uniques 1067; 2023-04-23 05:54:57,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=173, cellSize=7914, totalHeapSize=18232, min timestamp=1682229297395, max timestamp=1682229297402Num uniques 1067; 2023-04-23 05:54:57,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48852, new segment data size=9922, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-23 05:54:57,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48858, new segment data size=7914, suffix heap size=138936,new segment heap size=18232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=173 2023-04-23 05:54:57,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682229297420, max timestamp=1682229297428Num uniques 1076; 2023-04-23 05:54:57,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-23 05:54:57,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682229297420, max timestamp=1682229297428Num uniques 1076; 2023-04-23 05:54:57,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=10346, suffix heap size=139872,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=226 2023-04-23 05:54:57,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682229297420, max timestamp=1682229297428Num uniques 1032; 2023-04-23 05:54:57,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47262, new segment data size=10346, suffix heap size=135296,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1032, new segment cells count=226 2023-04-23 05:54:57,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682229297439, max timestamp=1682229297453Num uniques 1085; 2023-04-23 05:54:57,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-23 05:54:57,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682229297439, max timestamp=1682229297453Num uniques 1085; 2023-04-23 05:54:57,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-23 05:54:57,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682229297439, max timestamp=1682229297453Num uniques 1085; 2023-04-23 05:54:57,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-23 05:54:57,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682229297468, max timestamp=1682229297475Num uniques 1044; 2023-04-23 05:54:57,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-23 05:54:57,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682229297468, max timestamp=1682229297475Num uniques 1044; 2023-04-23 05:54:57,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-23 05:54:57,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1682229297468, max timestamp=1682229297475Num uniques 1044; 2023-04-23 05:54:57,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-23 05:54:57,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682229297495, max timestamp=1682229297505Num uniques 1103; 2023-04-23 05:54:57,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682229297495, max timestamp=1682229297505Num uniques 1103; 2023-04-23 05:54:57,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1682229297495, max timestamp=1682229297505Num uniques 1103; 2023-04-23 05:54:57,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-23 05:54:57,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-23 05:54:57,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-23 05:54:57,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682229297513, max timestamp=1682229297527Num uniques 1012; 2023-04-23 05:54:57,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-23 05:54:57,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1682229297513, max timestamp=1682229297527Num uniques 1012; 2023-04-23 05:54:57,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-23 05:54:57,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1682229297513, max timestamp=1682229297527Num uniques 1012; 2023-04-23 05:54:57,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-04-23 05:54:57,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1682229297533, max timestamp=1682229297548Num uniques 1121; 2023-04-23 05:54:57,550 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=10116, suffix heap size=144552,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=221 2023-04-23 05:54:57,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1682229297533, max timestamp=1682229297548Num uniques 1071; 2023-04-23 05:54:57,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-04-23 05:54:57,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682229297533, max timestamp=1682229297548Num uniques 1071; 2023-04-23 05:54:57,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=6870, suffix heap size=139352,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=150 2023-04-23 05:54:57,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682229297562, max timestamp=1682229297567Num uniques 1080; 2023-04-23 05:54:57,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682229297562, max timestamp=1682229297567Num uniques 1080; 2023-04-23 05:54:57,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1682229297562, max timestamp=1682229297567Num uniques 1009; 2023-04-23 05:54:57,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-23 05:54:57,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8240, suffix heap size=132904,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=180 2023-04-23 05:54:57,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-23 05:54:57,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682229297580, max timestamp=1682229297585Num uniques 1039; 2023-04-23 05:54:57,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-23 05:54:57,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682229297580, max timestamp=1682229297585Num uniques 1039; 2023-04-23 05:54:57,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1682229297580, max timestamp=1682229297585Num uniques 1039; 2023-04-23 05:54:57,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-23 05:54:57,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=10944, suffix heap size=136024,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=239 2023-04-23 05:54:57,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1682229297596, max timestamp=1682229297606Num uniques 1098; 2023-04-23 05:54:57,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1682229297596, max timestamp=1682229297606Num uniques 1098; 2023-04-23 05:54:57,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=13648, suffix heap size=142160,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=298 2023-04-23 05:54:57,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1682229297596, max timestamp=1682229297606Num uniques 1098; 2023-04-23 05:54:57,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=13648, suffix heap size=142160,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=298 2023-04-23 05:54:57,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=13648, suffix heap size=142160,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=298 2023-04-23 05:54:57,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229357623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229357624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229357625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229357624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229357627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229357728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229357728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229357729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229357729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229357731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,807 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/6a89a7ca8734473c90c6638a29bbcc91 2023-04-23 05:54:57,830 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/2f85da106031487faa506667dd269db0 2023-04-23 05:54:57,835 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/3059e55e5f48480693697a18c1ac9a43 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3059e55e5f48480693697a18c1ac9a43 2023-04-23 05:54:57,839 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3059e55e5f48480693697a18c1ac9a43, entries=150, sequenceid=167, filesize=11.8 K 2023-04-23 05:54:57,839 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/6a89a7ca8734473c90c6638a29bbcc91 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6a89a7ca8734473c90c6638a29bbcc91 2023-04-23 05:54:57,846 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6a89a7ca8734473c90c6638a29bbcc91, entries=150, sequenceid=167, filesize=11.8 K 2023-04-23 05:54:57,847 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/2f85da106031487faa506667dd269db0 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/2f85da106031487faa506667dd269db0 2023-04-23 05:54:57,853 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/2f85da106031487faa506667dd269db0, entries=150, sequenceid=167, filesize=11.8 K 2023-04-23 05:54:57,853 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=154.31 KB/158010 for 911c079a878d9ede8370cba6d74e3efd in 870ms, sequenceid=167, compaction requested=false 2023-04-23 05:54:57,854 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:57,854 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:57,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:57,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:57,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:57,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:57,855 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,855 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:57,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:57,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c0bd7f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:57,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:57,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:57,859 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,859 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:57,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:57,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:57,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:57,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:57,865 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:57,865 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:57,865 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:57,865 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:57,865 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:57,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:57,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:57,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:57,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:57,867 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:57,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:57,870 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,870 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:57,870 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:57,870 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:57,870 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:57,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:57,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,871 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:57,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:57,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:57,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:57,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:57,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:57,871 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:57,871 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:57,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:57,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:57,934 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=161.02 KB heapSize=452.34 KB 2023-04-23 05:54:57,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:54:57,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1682229297621, max timestamp=1682229297933Num uniques 1157; 2023-04-23 05:54:57,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=7185, suffix heap size=148296,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=157 2023-04-23 05:54:57,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:57,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1682229297621, max timestamp=1682229297933Num uniques 1157; 2023-04-23 05:54:57,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:54:57,935 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=7185, suffix heap size=148296,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=157 2023-04-23 05:54:57,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:54:57,936 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:54:57,936 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-23 05:54:57,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229357946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229357949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229357952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229357952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:57,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229357953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,957 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/75ea2035da8d4c6b9d2f8ad8ee5abd45 2023-04-23 05:54:57,971 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:57,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:57,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/6f15710bc080404dab1196b6765430bc 2023-04-23 05:54:57,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:57,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:57,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:57,975 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:57,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:57,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:57,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:57,976 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:57,976 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:57,977 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:57,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:57,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:57,978 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:57,978 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:57,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:57,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:57,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:57,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:57,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:57,980 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:57,980 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:57,980 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:57,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:57,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:57,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:57,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:57,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:57,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:57,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:57,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:57,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:57,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:57,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee20d28[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:57,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:57,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,997 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:57,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:57,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:57,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:57,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:57,999 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:58,002 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:58,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:58,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:58,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:58,002 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:58,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,005 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:58,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:58,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:58,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:58,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:58,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,009 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:58,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:58,009 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:58,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:58,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52786 deadline: 1682229358054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229358055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229358055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52824 deadline: 1682229358055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229358055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,109 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:58,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:58,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:58,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:58,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:58,116 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:58,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:58,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:58,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:58,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,118 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:58,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:58,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:58,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:58,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:58,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:58,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:58,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:58,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:58,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:58,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:58,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:58,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65f57727[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:58,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:58,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,125 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:58,125 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:58,125 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,125 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:58,125 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:58,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:58,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:58,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:58,130 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:58,131 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:58,131 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:58,131 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:58,131 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:58,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,134 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,135 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,139 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:58,139 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:58,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:58,139 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:58,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:58,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:58,139 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,140 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,140 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:58,140 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:58,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:58,240 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:58,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:58,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:58,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:58,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:58,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:58,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:58,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:58,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:58,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:58,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:58,247 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,247 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:58,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:58,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:58,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:58,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:58,250 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,250 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:58,250 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:58,250 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:58,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:58,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,251 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,252 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,252 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:58,253 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:58,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ec545d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:58,253 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:58,255 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,255 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:58,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:58,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:58,256 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:58,256 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:58,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:58,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:58,258 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:58,258 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:58,259 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:58,259 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:58,259 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:58,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229358259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229358259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229358259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229358260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229358260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,262 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,263 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:58,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:58,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:58,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,266 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:58,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:58,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:58,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:58,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:58,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:58,267 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:58,267 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:58,367 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:58,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:58,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:58,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:58,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:58,371 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:58,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:58,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:58,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:58,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,373 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,373 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:58,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:58,373 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,373 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:58,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:58,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:58,374 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:58,374 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:58,375 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,375 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:58,375 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:58,375 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:58,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:58,376 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,377 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,377 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,377 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:58,378 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:58,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@613028ba[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:58,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,381 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:58,381 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:58,381 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,381 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:58,381 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:58,383 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:58,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:58,385 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:58,385 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:58,385 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:58,385 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:58,385 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:58,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:58,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:58,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:58,388 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:58,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:58,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/ca60849574624bcf822bded65673c3a9 2023-04-23 05:54:58,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:58,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:58,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:58,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:58,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:58,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,393 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:58,394 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:58,393 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:58,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:58,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:58,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:58,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:58,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/75ea2035da8d4c6b9d2f8ad8ee5abd45 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/75ea2035da8d4c6b9d2f8ad8ee5abd45 2023-04-23 05:54:58,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/75ea2035da8d4c6b9d2f8ad8ee5abd45, entries=150, sequenceid=568, filesize=11.9 K 2023-04-23 05:54:58,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/6f15710bc080404dab1196b6765430bc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6f15710bc080404dab1196b6765430bc 2023-04-23 05:54:58,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6f15710bc080404dab1196b6765430bc, entries=150, sequenceid=568, filesize=11.9 K 2023-04-23 05:54:58,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/ca60849574624bcf822bded65673c3a9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca60849574624bcf822bded65673c3a9 2023-04-23 05:54:58,429 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca60849574624bcf822bded65673c3a9, entries=150, sequenceid=568, filesize=11.9 K 2023-04-23 05:54:58,430 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~78.27 KB/80150, heapSize ~214.94 KB/220096, currentSize=100.63 KB/103050 for 911c079a878d9ede8370cba6d74e3efd in 496ms, sequenceid=568, compaction requested=true 2023-04-23 05:54:58,430 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:58,430 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:58,430 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:58,430 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:58,430 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:58,430 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:58,435 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36361 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:58,435 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/A is initiating minor compaction (all files) 2023-04-23 05:54:58,435 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36361 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:58,435 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/B is initiating minor compaction (all files) 2023-04-23 05:54:58,435 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/A in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,435 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/B in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,435 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/e0d6d4797cb8452bba1215755d0b1766, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3059e55e5f48480693697a18c1ac9a43, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/75ea2035da8d4c6b9d2f8ad8ee5abd45] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=35.5 K 2023-04-23 05:54:58,435 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/e515e9c08b2a432f925462896f08f187, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6a89a7ca8734473c90c6638a29bbcc91, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6f15710bc080404dab1196b6765430bc] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=35.5 K 2023-04-23 05:54:58,435 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e0d6d4797cb8452bba1215755d0b1766, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682229296073 2023-04-23 05:54:58,436 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e515e9c08b2a432f925462896f08f187, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682229296073 2023-04-23 05:54:58,436 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 3059e55e5f48480693697a18c1ac9a43, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682229296402 2023-04-23 05:54:58,436 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 6a89a7ca8734473c90c6638a29bbcc91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682229296402 2023-04-23 05:54:58,436 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 75ea2035da8d4c6b9d2f8ad8ee5abd45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1682229297621 2023-04-23 05:54:58,436 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 6f15710bc080404dab1196b6765430bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1682229297621 2023-04-23 05:54:58,444 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#B#compaction#210 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:54:58,444 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#A#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:54:58,494 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:58,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:58,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:58,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:58,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:58,498 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:58,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:58,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:58,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:58,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:58,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:58,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:58,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:58,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:58,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:58,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:58,502 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,502 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:58,503 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-23 05:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:54:58,503 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:58,564 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:58,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:58,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229358571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229358571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229358572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229358572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229358574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229358676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229358676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229358676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229358676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229358677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,702 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-23 05:54:58,862 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/f19082223ea94e0a9394c337f74f7778 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f19082223ea94e0a9394c337f74f7778 2023-04-23 05:54:58,862 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/fd218d14a9414d62ad579c16f6d44ee2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/fd218d14a9414d62ad579c16f6d44ee2 2023-04-23 05:54:58,867 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/B of 911c079a878d9ede8370cba6d74e3efd into f19082223ea94e0a9394c337f74f7778(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:58,867 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:58,867 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/B, priority=13, startTime=1682229298430; duration=0sec 2023-04-23 05:54:58,867 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:54:58,867 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:54:58,868 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/A of 911c079a878d9ede8370cba6d74e3efd into fd218d14a9414d62ad579c16f6d44ee2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:58,868 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:58,868 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/A, priority=13, startTime=1682229298430; duration=0sec 2023-04-23 05:54:58,868 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:58,868 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36361 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:54:58,869 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/C is initiating minor compaction (all files) 2023-04-23 05:54:58,869 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/C in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:58,869 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/031f33cc2ec0496cbfcf7ef43421ff4c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/2f85da106031487faa506667dd269db0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca60849574624bcf822bded65673c3a9] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=35.5 K 2023-04-23 05:54:58,869 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 031f33cc2ec0496cbfcf7ef43421ff4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682229296073 2023-04-23 05:54:58,870 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2f85da106031487faa506667dd269db0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1682229296402 2023-04-23 05:54:58,870 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ca60849574624bcf822bded65673c3a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1682229297621 2023-04-23 05:54:58,881 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#C#compaction#212 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:54:58,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229358882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229358882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229358882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229358882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:58,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229358882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:58,918 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/52bf6627e70c4b5786dd05dae8223f5a 2023-04-23 05:54:58,931 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/2cf3ea15aa944db19b1cab8f364d7900 2023-04-23 05:54:58,970 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/7b7fd4c83ad14515bdb442750a6bb6cf 2023-04-23 05:54:58,975 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/52bf6627e70c4b5786dd05dae8223f5a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/52bf6627e70c4b5786dd05dae8223f5a 2023-04-23 05:54:58,986 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/52bf6627e70c4b5786dd05dae8223f5a, entries=150, sequenceid=586, filesize=11.9 K 2023-04-23 05:54:58,987 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/2cf3ea15aa944db19b1cab8f364d7900 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cf3ea15aa944db19b1cab8f364d7900 2023-04-23 05:54:58,992 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cf3ea15aa944db19b1cab8f364d7900, entries=150, sequenceid=586, filesize=11.9 K 2023-04-23 05:54:58,993 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/7b7fd4c83ad14515bdb442750a6bb6cf as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7b7fd4c83ad14515bdb442750a6bb6cf 2023-04-23 05:54:59,002 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7b7fd4c83ad14515bdb442750a6bb6cf, entries=150, sequenceid=586, filesize=11.9 K 2023-04-23 05:54:59,003 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 911c079a878d9ede8370cba6d74e3efd in 501ms, sequenceid=586, compaction requested=false 2023-04-23 05:54:59,003 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:59,003 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:54:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:54:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:54:59,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:54:59,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:59,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:59,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:59,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:59,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:54:59,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fc5e2f9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:54:59,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:54:59,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:59,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:59,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:54:59,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:54:59,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:54:59,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:54:59,010 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:54:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:59,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:59,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:54:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:54:59,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:54:59,013 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:54:59,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:54:59,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:54:59,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:54:59,013 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:54:59,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:59,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:59,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:54:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:54:59,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:59,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:54:59,018 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:54:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,019 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:54:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:54:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:59,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:54:59,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:59,024 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:54:59,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:59,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:59,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:54:59,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:54:59,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:54:59,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:59,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:54:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:59,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:59,025 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:54:59,025 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:54:59,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:54:59,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:54:59,125 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:54:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:54:59,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:54:59,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:54:59,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:54:59,128 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:54:59,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:59,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:54:59,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:54:59,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:59,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:59,130 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:54:59,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:54:59,131 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,131 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:59,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:54:59,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:54:59,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:54:59,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:54:59,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:54:59,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:54:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:54:59,134 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:54:59,134 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:54:59,135 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-23 05:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:54:59,135 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:54:59,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:54:59,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:54:59,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229359200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229359203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229359204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229359204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229359204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,299 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/23cd8a6d424f4adfbeb0a1cc0f0c5117 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/23cd8a6d424f4adfbeb0a1cc0f0c5117 2023-04-23 05:54:59,304 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/C of 911c079a878d9ede8370cba6d74e3efd into 23cd8a6d424f4adfbeb0a1cc0f0c5117(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:54:59,304 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:54:59,304 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/C, priority=13, startTime=1682229298430; duration=0sec 2023-04-23 05:54:59,304 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:54:59,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229359305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229359307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229359309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229359309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229359309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229359508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229359508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229359513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229359513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229359513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,546 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/83bbe7dde6204f219db5471592705063 2023-04-23 05:54:59,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229359811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229359812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229359816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229359817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:54:59,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229359817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:54:59,960 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/2cedf91eb0ff4d7485ea0e6aeac74aa8 2023-04-23 05:55:00,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:00,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229360313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:00,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229360314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:00,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229360320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:00,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229360321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:00,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229360322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,371 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=602 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/a0aa8f11cf484f37863b0f0334715fb4 2023-04-23 05:55:00,376 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/83bbe7dde6204f219db5471592705063 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/83bbe7dde6204f219db5471592705063 2023-04-23 05:55:00,380 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/83bbe7dde6204f219db5471592705063, entries=150, sequenceid=602, filesize=11.9 K 2023-04-23 05:55:00,381 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/2cedf91eb0ff4d7485ea0e6aeac74aa8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cedf91eb0ff4d7485ea0e6aeac74aa8 2023-04-23 05:55:00,385 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cedf91eb0ff4d7485ea0e6aeac74aa8, entries=150, sequenceid=602, filesize=11.9 K 2023-04-23 05:55:00,386 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/a0aa8f11cf484f37863b0f0334715fb4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a0aa8f11cf484f37863b0f0334715fb4 2023-04-23 05:55:00,389 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a0aa8f11cf484f37863b0f0334715fb4, entries=150, sequenceid=602, filesize=11.9 K 2023-04-23 05:55:00,390 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 911c079a878d9ede8370cba6d74e3efd in 1255ms, sequenceid=602, compaction requested=true 2023-04-23 05:55:00,390 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:00,390 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:00,391 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:00,392 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:00,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,393 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:00,394 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,394 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:00,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:00,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5147e269[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:00,396 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:00,396 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,398 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,398 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:00,398 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:00,398 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:00,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:00,399 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:00,403 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:00,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:00,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:00,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:00,403 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:00,405 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,405 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:00,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,405 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:00,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,406 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:00,407 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:00,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:00,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:00,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:00,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:00,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:00,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:00,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:00,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:00,411 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:00,411 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:00,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:00,511 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:00,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:00,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:00,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:00,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:00,515 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:00,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:00,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:00,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:00,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:00,517 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:00,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:00,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:00,518 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,518 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:00,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:00,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:00,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:00,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:00,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:00,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:00,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:00,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:00,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:00,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:00,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:00,519 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-23 05:55:00,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:00,520 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:00,520 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:00,520 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:00,520 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:00,520 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:00,527 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/b95d0f9976ed4b4ebfc324290f331e56 2023-04-23 05:55:00,938 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/7249124d98d44bc5b81a6699024284d6 2023-04-23 05:55:00,949 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/51de38dfd08644eaaecadc56b0e0b49c 2023-04-23 05:55:00,953 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/b95d0f9976ed4b4ebfc324290f331e56 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b95d0f9976ed4b4ebfc324290f331e56 2023-04-23 05:55:00,957 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b95d0f9976ed4b4ebfc324290f331e56, entries=150, sequenceid=622, filesize=11.9 K 2023-04-23 05:55:00,958 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/7249124d98d44bc5b81a6699024284d6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7249124d98d44bc5b81a6699024284d6 2023-04-23 05:55:00,961 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7249124d98d44bc5b81a6699024284d6, entries=150, sequenceid=622, filesize=11.9 K 2023-04-23 05:55:00,962 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/51de38dfd08644eaaecadc56b0e0b49c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/51de38dfd08644eaaecadc56b0e0b49c 2023-04-23 05:55:00,978 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/51de38dfd08644eaaecadc56b0e0b49c, entries=150, sequenceid=622, filesize=11.9 K 2023-04-23 05:55:00,978 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 911c079a878d9ede8370cba6d74e3efd in 459ms, sequenceid=622, compaction requested=true 2023-04-23 05:55:00,978 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:00,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:00,980 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,980 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25fd0aa0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:00,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:00,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,983 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:00,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:00,984 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,984 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:00,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:00,985 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:00,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:00,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:00,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:00,988 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:00,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:00,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:00,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:00,988 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:00,993 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,993 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:00,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,993 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:00,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:00,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:00,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:00,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:00,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:00,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:00,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:00,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:00,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:00,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:00,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:00,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:00,998 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:00,998 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:00,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,098 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:01,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:01,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:01,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:01,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:01,102 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:01,102 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,102 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:01,102 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:01,103 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,103 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,104 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:01,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:01,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:01,106 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,106 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:01,106 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:01,106 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:01,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,108 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4568835d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:01,110 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:01,110 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,111 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:01,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:01,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:01,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:01,116 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:01,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:01,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:01,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:01,116 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:01,117 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,117 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,117 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,119 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:01,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:01,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,122 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:01,122 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:01,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,222 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:01,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:01,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:01,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:01,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:01,226 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:01,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:01,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:01,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:01,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:01,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:01,233 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,233 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:01,233 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:01,233 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:01,234 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:01,235 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:01,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@758a6323[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:01,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:01,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:01,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:01,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:01,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:01,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:01,243 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:01,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:01,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:01,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:01,243 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:01,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,247 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,249 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:01,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:01,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,250 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:01,250 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:01,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:01,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:01,329 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:01,329 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:01,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:01,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:01,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:01,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:01,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:01,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/54e71d6b634245118abff43f90afb512 2023-04-23 05:55:01,350 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:01,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:01,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:01,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:01,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:01,356 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:01,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:01,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:01,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,358 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:01,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:01,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:01,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:01,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:01,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:01,360 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:01,360 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:01,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:01,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:01,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:01,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,364 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,365 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:01,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b308351[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:01,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:01,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,367 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:01,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:01,367 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,367 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:01,368 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:01,369 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,369 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:01,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:01,371 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:01,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:01,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:01,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:01,371 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:01,372 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,372 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,372 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,374 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229361348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229361349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229301340, max timestamp=1682229301352Num uniques 859; 2023-04-23 05:55:01,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229301340, max timestamp=1682229301352Num uniques 859; 2023-04-23 05:55:01,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:01,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:01,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229301340, max timestamp=1682229301352Num uniques 859; 2023-04-23 05:55:01,376 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:01,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,377 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:01,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:01,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,377 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:01,377 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:01,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229301397, max timestamp=1682229301404Num uniques 1018; 2023-04-23 05:55:01,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229301397, max timestamp=1682229301404Num uniques 1018; 2023-04-23 05:55:01,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:55:01,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229361399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:55:01,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229301397, max timestamp=1682229301404Num uniques 1018; 2023-04-23 05:55:01,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:55:01,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229361448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:52816 deadline: 1682229361451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,477 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:01,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:01,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229361477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229361477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:01,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:01,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:01,481 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:01,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:01,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:01,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,482 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,482 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:01,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:01,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:01,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:01,485 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,485 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:01,485 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:01,485 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:01,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:01,486 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,486 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:01,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5655229d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:01,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:01,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,489 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,489 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:01,489 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:01,489 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,490 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:01,489 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:01,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:01,492 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:01,494 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:01,494 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:01,494 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:01,495 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:01,495 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:01,496 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,496 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,496 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:01,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:01,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,501 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:01,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:01,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,501 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,501 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,501 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:01,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229361507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229361552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52816 deadline: 1682229361555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,601 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:01,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:01,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:01,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:01,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:01,605 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:01,605 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,605 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:01,605 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:01,605 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,605 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,606 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,606 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:01,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:01,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:01,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:01,608 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,608 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:01,608 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:01,608 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:01,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:01,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:01,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:01,611 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,611 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,612 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:01,613 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:01,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57a62161[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:01,613 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,617 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,617 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:01,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:01,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:01,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:01,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:01,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:01,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:01,621 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:01,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:01,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:01,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:01,621 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:01,623 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,623 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,623 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:01,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:01,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,629 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,629 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,629 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:01,629 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:01,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,629 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:01,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229361679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229361679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229361710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,729 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:01,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:01,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:01,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:01,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:01,732 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:01,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:01,733 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:01,733 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,733 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,734 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:01,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:01,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:01,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:01,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:01,735 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:01,736 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,736 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:01,736 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:01,736 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:01,738 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,738 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,738 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,738 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:01,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:01,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7193c3cd[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:01,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,740 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:01,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:01,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:01,741 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,741 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:01,743 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,743 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:01,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:01,746 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:01,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:01,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:01,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:01,746 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:01,747 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,747 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,747 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,751 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,751 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,751 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:01,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:01,751 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,752 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:01,752 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:01,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/c1090617b5c8420aa8897c399c78b44b 2023-04-23 05:55:01,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229361756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52816 deadline: 1682229361759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,853 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:01,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:01,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:01,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:01,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:01,856 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:01,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:01,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:01,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,858 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,858 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,858 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:01,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:01,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:01,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:01,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:01,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:01,860 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,860 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:01,860 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:01,860 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:01,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:01,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,863 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:01,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:01,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e26fca5[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:01,865 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:01,865 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,866 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:01,866 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:01,866 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:01,866 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,866 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:01,867 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,867 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:01,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,868 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,868 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:01,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:01,870 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:01,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:01,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:01,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:01,870 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:01,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,872 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:01,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,877 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,877 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:01,877 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:01,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,977 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:01,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:01,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:01,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:01,980 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:01,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:01,980 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:01,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,982 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:01,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:01,983 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,983 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:01,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:01,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:01,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:01,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:01,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:01,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229361981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:01,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:01,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:01,984 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:01,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:01,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:01,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:01,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:01,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:01,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:01,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229361983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,986 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,986 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,986 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,986 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:01,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11187a36[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:01,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:01,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:01,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:01,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:01,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:01,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:01,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:01,997 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:01,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:01,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:01,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:01,997 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:01,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:01,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:01,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:01,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:01,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:02,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:02,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:02,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:02,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:02,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:02,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:02,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:02,003 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:02,003 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:02,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:02,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229362013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229362062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52816 deadline: 1682229362064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,103 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:02,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:02,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:02,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:02,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:02,107 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:02,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:02,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:02,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:02,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:02,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:02,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:02,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:02,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:02,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:02,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:02,115 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,115 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:02,115 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:02,115 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:02,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:02,117 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,117 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:02,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d3fc5d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:02,119 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:02,119 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,120 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,121 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:02,121 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:02,121 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:02,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:02,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:02,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:02,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:02,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:02,124 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:02,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:02,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:02,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:02,124 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:02,127 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,127 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,127 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,127 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:02,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:02,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,131 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,131 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,131 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:02,131 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,131 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:02,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:02,131 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:02,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:02,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:02,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:02,131 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,132 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:02,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:02,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:02,132 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:02,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:02,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/17ecbcceb2fc4705a7a5585bbdf0f6bc 2023-04-23 05:55:02,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/54e71d6b634245118abff43f90afb512 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/54e71d6b634245118abff43f90afb512 2023-04-23 05:55:02,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/54e71d6b634245118abff43f90afb512, entries=150, sequenceid=634, filesize=11.9 K 2023-04-23 05:55:02,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/c1090617b5c8420aa8897c399c78b44b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/c1090617b5c8420aa8897c399c78b44b 2023-04-23 05:55:02,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/c1090617b5c8420aa8897c399c78b44b, entries=150, sequenceid=634, filesize=11.9 K 2023-04-23 05:55:02,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/17ecbcceb2fc4705a7a5585bbdf0f6bc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/17ecbcceb2fc4705a7a5585bbdf0f6bc 2023-04-23 05:55:02,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/17ecbcceb2fc4705a7a5585bbdf0f6bc, entries=150, sequenceid=634, filesize=11.9 K 2023-04-23 05:55:02,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 911c079a878d9ede8370cba6d74e3efd in 863ms, sequenceid=634, compaction requested=true 2023-04-23 05:55:02,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:02,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:02,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-23 05:55:02,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-23 05:55:02,192 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:55:02,193 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:55:02,195 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61136 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:55:02,195 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61136 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:55:02,195 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/A is initiating minor compaction (all files) 2023-04-23 05:55:02,195 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/C is initiating minor compaction (all files) 2023-04-23 05:55:02,195 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/A in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,195 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/C in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,195 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/fd218d14a9414d62ad579c16f6d44ee2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/52bf6627e70c4b5786dd05dae8223f5a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/83bbe7dde6204f219db5471592705063, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b95d0f9976ed4b4ebfc324290f331e56, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/54e71d6b634245118abff43f90afb512] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=59.7 K 2023-04-23 05:55:02,195 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/23cd8a6d424f4adfbeb0a1cc0f0c5117, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7b7fd4c83ad14515bdb442750a6bb6cf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a0aa8f11cf484f37863b0f0334715fb4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/51de38dfd08644eaaecadc56b0e0b49c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/17ecbcceb2fc4705a7a5585bbdf0f6bc] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=59.7 K 2023-04-23 05:55:02,196 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 23cd8a6d424f4adfbeb0a1cc0f0c5117, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1682229297621 2023-04-23 05:55:02,196 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting fd218d14a9414d62ad579c16f6d44ee2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1682229297621 2023-04-23 05:55:02,196 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7b7fd4c83ad14515bdb442750a6bb6cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682229297950 2023-04-23 05:55:02,197 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 52bf6627e70c4b5786dd05dae8223f5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682229297950 2023-04-23 05:55:02,197 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a0aa8f11cf484f37863b0f0334715fb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682229298570 2023-04-23 05:55:02,197 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 83bbe7dde6204f219db5471592705063, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682229298570 2023-04-23 05:55:02,197 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 51de38dfd08644eaaecadc56b0e0b49c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682229299197 2023-04-23 05:55:02,197 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b95d0f9976ed4b4ebfc324290f331e56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682229299197 2023-04-23 05:55:02,198 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 17ecbcceb2fc4705a7a5585bbdf0f6bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1682229301327 2023-04-23 05:55:02,198 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 54e71d6b634245118abff43f90afb512, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1682229301327 2023-04-23 05:55:02,210 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#A#compaction#224 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:02,215 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#C#compaction#225 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:02,232 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:02,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:02,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:02,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:02,240 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:02,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:02,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:02,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:02,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:02,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:02,244 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/1bcc6505af704cd0bfe4a3b2f88cc330 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/1bcc6505af704cd0bfe4a3b2f88cc330 2023-04-23 05:55:02,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:02,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:02,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:02,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:02,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:02,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:02,246 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/924846bd3bc94cfbb0f6154a690f8798 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/924846bd3bc94cfbb0f6154a690f8798 2023-04-23 05:55:02,250 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/C of 911c079a878d9ede8370cba6d74e3efd into 1bcc6505af704cd0bfe4a3b2f88cc330(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:02,250 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:02,250 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/C, priority=11, startTime=1682229302192; duration=0sec 2023-04-23 05:55:02,250 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:02,250 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:55:02,252 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:02,252 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:02,253 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61136 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:55:02,253 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/B is initiating minor compaction (all files) 2023-04-23 05:55:02,253 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/B in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,253 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f19082223ea94e0a9394c337f74f7778, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cf3ea15aa944db19b1cab8f364d7900, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cedf91eb0ff4d7485ea0e6aeac74aa8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7249124d98d44bc5b81a6699024284d6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/c1090617b5c8420aa8897c399c78b44b] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=59.7 K 2023-04-23 05:55:02,254 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f19082223ea94e0a9394c337f74f7778, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1682229297621 2023-04-23 05:55:02,254 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2cf3ea15aa944db19b1cab8f364d7900, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1682229297950 2023-04-23 05:55:02,254 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2cedf91eb0ff4d7485ea0e6aeac74aa8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=602, earliestPutTs=1682229298570 2023-04-23 05:55:02,255 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7249124d98d44bc5b81a6699024284d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682229299197 2023-04-23 05:55:02,256 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,256 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c1090617b5c8420aa8897c399c78b44b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1682229301327 2023-04-23 05:55:02,256 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:02,256 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=134.18 KB heapSize=384.14 KB 2023-04-23 05:55:02,257 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:02,257 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:02,257 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:02,257 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:02,257 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:02,257 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:02,265 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/A of 911c079a878d9ede8370cba6d74e3efd into 924846bd3bc94cfbb0f6154a690f8798(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:02,266 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:02,266 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/A, priority=11, startTime=1682229302192; duration=0sec 2023-04-23 05:55:02,266 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:02,269 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#B#compaction#226 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:02,285 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=689 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/f9ea77137d8c47d49221749b0b1856fc 2023-04-23 05:55:02,297 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=689 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/457184c99f174749a13a187c8c62260b 2023-04-23 05:55:02,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:02,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:02,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229362498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229362500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229362519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229362569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52816 deadline: 1682229362570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229362601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229362603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,687 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/2481978c26274a18bba24392cb9ccf72 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2481978c26274a18bba24392cb9ccf72 2023-04-23 05:55:02,691 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/B of 911c079a878d9ede8370cba6d74e3efd into 2481978c26274a18bba24392cb9ccf72(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:02,691 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:02,692 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/B, priority=11, startTime=1682229302192; duration=0sec 2023-04-23 05:55:02,692 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:02,708 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=689 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/9ebfacec72034f00af6e733b18404a50 2023-04-23 05:55:02,712 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/f9ea77137d8c47d49221749b0b1856fc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f9ea77137d8c47d49221749b0b1856fc 2023-04-23 05:55:02,716 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f9ea77137d8c47d49221749b0b1856fc, entries=150, sequenceid=689, filesize=11.9 K 2023-04-23 05:55:02,717 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/457184c99f174749a13a187c8c62260b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/457184c99f174749a13a187c8c62260b 2023-04-23 05:55:02,721 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/457184c99f174749a13a187c8c62260b, entries=150, sequenceid=689, filesize=11.9 K 2023-04-23 05:55:02,722 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/9ebfacec72034f00af6e733b18404a50 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9ebfacec72034f00af6e733b18404a50 2023-04-23 05:55:02,728 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9ebfacec72034f00af6e733b18404a50, entries=150, sequenceid=689, filesize=11.9 K 2023-04-23 05:55:02,729 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.09 KB/393312, currentSize=46.96 KB/48090 for 911c079a878d9ede8370cba6d74e3efd in 473ms, sequenceid=689, compaction requested=false 2023-04-23 05:55:02,729 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:02,729 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:02,729 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,732 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:02,732 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:02,733 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,733 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:02,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1831d4c1[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:02,736 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:02,736 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,739 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:02,739 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:02,739 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:02,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:02,740 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:02,740 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:02,740 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:02,740 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:02,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:02,743 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:02,743 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:02,743 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:02,743 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:02,743 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:02,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:02,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,751 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:02,751 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:02,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:02,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:02,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:02,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:02,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:02,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,753 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,753 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,753 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,753 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:02,753 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:02,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:02,807 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:55:02,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:02,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:02,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:02,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:02,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:02,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:02,816 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/b2811b93928849d2bdaeb18cfc7f0435 2023-04-23 05:55:02,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229362838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229302830, max timestamp=1682229302839Num uniques 859; 2023-04-23 05:55:02,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:02,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229302830, max timestamp=1682229302839Num uniques 859; 2023-04-23 05:55:02,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:02,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229302830, max timestamp=1682229302839Num uniques 859; 2023-04-23 05:55:02,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:02,853 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:02,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:02,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:02,856 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:02,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:02,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:02,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:02,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,860 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:02,860 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:02,860 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,860 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:02,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:02,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:02,862 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,863 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:02,863 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:02,863 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:02,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:02,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,865 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:02,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41abc0b6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:02,867 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:02,867 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,868 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:02,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:02,869 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,869 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:02,869 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,869 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:02,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:02,872 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:02,872 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:02,872 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:02,872 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:02,872 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:02,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:02,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:02,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:02,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:02,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:02,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:02,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:02,881 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:02,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:02,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,881 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:02,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:02,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,882 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:02,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229362895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229362941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,982 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:02,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:02,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:02,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:02,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:02,986 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:02,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:02,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:02,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:02,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:02,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:02,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:02,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:02,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:02,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:02,990 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:02,990 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:02,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13b0cb7f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:02,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:02,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:02,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:02,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:02,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:02,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:02,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:02,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:02,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:02,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:02,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229362998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:03,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:03,001 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:03,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:03,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:03,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:03,001 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:03,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,004 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:03,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:03,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:03,007 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:03,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,007 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:03,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,107 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:03,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:03,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:03,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:03,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:03,111 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:03,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:03,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:03,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:03,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:03,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:03,116 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,116 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:03,116 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:03,116 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:03,118 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:03,119 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b69fc40[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:03,119 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,120 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:03,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,121 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:03,121 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:03,121 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:03,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:03,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:03,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:03,125 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:03,125 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:03,125 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:03,125 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:03,125 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:03,127 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,127 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:03,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:03,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,132 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:03,133 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:03,132 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,133 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,133 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:03,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229363145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:03,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229363200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/2eda4e2be29c4dec9855022c582457cb 2023-04-23 05:55:03,233 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:03,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:03,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:03,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:03,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:03,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:03,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:03,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:03,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,246 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:03,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:03,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:03,248 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,248 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:03,248 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:03,248 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:03,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d68b830[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:03,252 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:03,252 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,253 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,254 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:03,254 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:03,254 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,254 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:03,254 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:03,255 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,256 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:03,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:03,258 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:03,258 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:03,258 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:03,258 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:03,258 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:03,260 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,260 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,260 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,261 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,262 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:03,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:03,268 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,268 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:03,268 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:03,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,368 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:03,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:03,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:03,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:03,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:03,372 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:03,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:03,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:03,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,373 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,374 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,374 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,374 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,374 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:03,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:03,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:03,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:03,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,375 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:03,376 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,376 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:03,376 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:03,376 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:03,377 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,378 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,378 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:03,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72d97b5b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:03,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:03,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,381 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:03,381 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:03,381 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:03,382 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,382 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:03,383 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:03,386 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:03,386 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:03,386 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:03,386 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:03,386 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:03,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,388 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,391 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,391 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,391 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:03,391 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:03,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,391 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,391 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,392 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:03,392 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:03,391 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:03,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:03,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229363449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,492 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:03,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:03,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:03,495 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,499 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,499 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,499 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,499 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:03,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:03,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:03,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:03,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:03,501 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,501 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:03,501 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:03,501 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:03,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:03,502 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,503 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,503 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6727bd4c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:03,504 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:03,505 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,506 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:03,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:03,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229363504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:03,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:03,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:03,510 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:03,510 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:03,510 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:03,510 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:03,510 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:03,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,517 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,517 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,517 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:03,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:03,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,518 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,518 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:03,518 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:03,517 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:03,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229363525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:03,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229363573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:03,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52816 deadline: 1682229363576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,618 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:03,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:03,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:03,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:03,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:03,621 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:03,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,622 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:03,622 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:03,622 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,622 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,623 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:03,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:03,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:03,625 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,625 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:03,625 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:03,625 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:03,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:03,627 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:03,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5398d614[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:03,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:03,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,630 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:03,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:03,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:03,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:03,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:03,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:03,634 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:03,634 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:03,635 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:03,635 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:03,635 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:03,636 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,636 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:03,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,636 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:03,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,637 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,638 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:03,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:03,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:03,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:03,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,640 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:03,640 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:03,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:03,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:03,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:03,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:03,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:03,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/78c75f9332bf42be9778e1feecd641ca 2023-04-23 05:55:03,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/b2811b93928849d2bdaeb18cfc7f0435 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2811b93928849d2bdaeb18cfc7f0435 2023-04-23 05:55:03,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2811b93928849d2bdaeb18cfc7f0435, entries=150, sequenceid=702, filesize=11.9 K 2023-04-23 05:55:03,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/2eda4e2be29c4dec9855022c582457cb as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2eda4e2be29c4dec9855022c582457cb 2023-04-23 05:55:03,656 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2eda4e2be29c4dec9855022c582457cb, entries=150, sequenceid=702, filesize=11.9 K 2023-04-23 05:55:03,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/78c75f9332bf42be9778e1feecd641ca as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/78c75f9332bf42be9778e1feecd641ca 2023-04-23 05:55:03,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/78c75f9332bf42be9778e1feecd641ca, entries=150, sequenceid=702, filesize=11.9 K 2023-04-23 05:55:03,661 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 911c079a878d9ede8370cba6d74e3efd in 854ms, sequenceid=702, compaction requested=true 2023-04-23 05:55:03,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:03,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:03,662 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:03,662 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:03,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:03,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:03,663 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:03,663 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:03,663 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/A is initiating minor compaction (all files) 2023-04-23 05:55:03,663 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/B is initiating minor compaction (all files) 2023-04-23 05:55:03,663 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/A in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,663 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/B in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,663 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/924846bd3bc94cfbb0f6154a690f8798, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f9ea77137d8c47d49221749b0b1856fc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2811b93928849d2bdaeb18cfc7f0435] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=36.1 K 2023-04-23 05:55:03,663 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2481978c26274a18bba24392cb9ccf72, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/457184c99f174749a13a187c8c62260b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2eda4e2be29c4dec9855022c582457cb] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=36.1 K 2023-04-23 05:55:03,663 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 924846bd3bc94cfbb0f6154a690f8798, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1682229301327 2023-04-23 05:55:03,664 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2481978c26274a18bba24392cb9ccf72, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1682229301327 2023-04-23 05:55:03,664 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting f9ea77137d8c47d49221749b0b1856fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=689, earliestPutTs=1682229301419 2023-04-23 05:55:03,664 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 457184c99f174749a13a187c8c62260b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=689, earliestPutTs=1682229301419 2023-04-23 05:55:03,664 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b2811b93928849d2bdaeb18cfc7f0435, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1682229302492 2023-04-23 05:55:03,665 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2eda4e2be29c4dec9855022c582457cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1682229302492 2023-04-23 05:55:03,676 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#B#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:03,677 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#A#compaction#234 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:03,691 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/a704432cb4b5443b8a7d91fcff0d394b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/a704432cb4b5443b8a7d91fcff0d394b 2023-04-23 05:55:03,694 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/8683167810e04f849fd7cd492a500793 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/8683167810e04f849fd7cd492a500793 2023-04-23 05:55:03,701 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/B of 911c079a878d9ede8370cba6d74e3efd into 8683167810e04f849fd7cd492a500793(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:03,701 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/A of 911c079a878d9ede8370cba6d74e3efd into a704432cb4b5443b8a7d91fcff0d394b(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:03,701 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:03,701 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:03,701 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/A, priority=13, startTime=1682229303662; duration=0sec 2023-04-23 05:55:03,701 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/B, priority=13, startTime=1682229303662; duration=0sec 2023-04-23 05:55:03,701 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:03,701 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:03,701 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:03,702 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:03,702 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/C is initiating minor compaction (all files) 2023-04-23 05:55:03,702 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/C in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,702 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/1bcc6505af704cd0bfe4a3b2f88cc330, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9ebfacec72034f00af6e733b18404a50, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/78c75f9332bf42be9778e1feecd641ca] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=36.1 K 2023-04-23 05:55:03,703 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 1bcc6505af704cd0bfe4a3b2f88cc330, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1682229301327 2023-04-23 05:55:03,703 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 9ebfacec72034f00af6e733b18404a50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=689, earliestPutTs=1682229301419 2023-04-23 05:55:03,704 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 78c75f9332bf42be9778e1feecd641ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1682229302492 2023-04-23 05:55:03,709 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#C#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:03,740 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:03,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:03,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:03,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:03,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:03,744 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:03,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:03,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:03,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:03,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:03,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:03,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:03,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:03,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:03,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:03,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:03,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:03,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:03,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:03,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:03,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:03,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:03,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:03,751 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-23 05:55:03,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:03,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:03,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:03,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:03,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:03,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:03,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:03,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:03,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:03,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229363975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:04,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:04,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229364008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:04,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:04,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229364078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:04,118 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/410dfa6905934fafb240c5adfaf70203 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/410dfa6905934fafb240c5adfaf70203 2023-04-23 05:55:04,123 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/C of 911c079a878d9ede8370cba6d74e3efd into 410dfa6905934fafb240c5adfaf70203(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:04,123 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:04,123 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/C, priority=13, startTime=1682229303662; duration=0sec 2023-04-23 05:55:04,123 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:04,160 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/dde6f51a6a874435beb7896f2fb9527e 2023-04-23 05:55:04,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:04,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229364281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:04,580 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/3e5e9cb7e9a84cfaa003c9cfeb805d87 2023-04-23 05:55:04,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:04,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229364585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:04,997 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/a5e72b1c54b44567acab6eb1d27b7140 2023-04-23 05:55:05,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/dde6f51a6a874435beb7896f2fb9527e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/dde6f51a6a874435beb7896f2fb9527e 2023-04-23 05:55:05,006 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/dde6f51a6a874435beb7896f2fb9527e, entries=150, sequenceid=740, filesize=11.9 K 2023-04-23 05:55:05,007 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/3e5e9cb7e9a84cfaa003c9cfeb805d87 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/3e5e9cb7e9a84cfaa003c9cfeb805d87 2023-04-23 05:55:05,011 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/3e5e9cb7e9a84cfaa003c9cfeb805d87, entries=150, sequenceid=740, filesize=11.9 K 2023-04-23 05:55:05,012 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/a5e72b1c54b44567acab6eb1d27b7140 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a5e72b1c54b44567acab6eb1d27b7140 2023-04-23 05:55:05,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:05,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229365015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,016 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a5e72b1c54b44567acab6eb1d27b7140, entries=150, sequenceid=740, filesize=11.9 K 2023-04-23 05:55:05,017 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 911c079a878d9ede8370cba6d74e3efd in 1266ms, sequenceid=740, compaction requested=false 2023-04-23 05:55:05,017 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:05,017 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,017 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:05,017 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:05,017 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,017 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:05,017 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:05,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,021 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,021 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:05,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b995d6f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:05,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:05,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,023 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:05,024 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,024 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:05,025 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:05,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:05,028 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:05,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:05,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:05,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:05,028 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:05,030 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,030 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,030 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,031 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,032 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,035 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:05,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:05,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:05,036 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:05,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,036 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:05,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:05,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:55:05,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:05,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:05,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:05,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:05,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:05,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:05,136 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:05,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:05,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:05,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:05,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:05,139 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:05,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:05,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:05,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,141 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,141 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,141 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,141 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:05,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:05,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:05,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:05,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:05,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:05,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:05,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:05,143 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,143 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:05,143 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:05,143 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:05,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:05,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:05,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:05,144 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,145 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,145 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:05,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a3f2e4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:05,146 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:05,146 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:05,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229365145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,148 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,148 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:05,148 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:05,148 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:05,148 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,148 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:05,150 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,150 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:05,150 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,150 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:05,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:05,153 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:05,153 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:05,153 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:05,153 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:05,153 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:05,154 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,154 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,154 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,158 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:05,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,159 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:05,159 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:05,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:05,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:05,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229365248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,259 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:05,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:05,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:05,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:05,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:05,262 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:05,262 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,262 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:05,262 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:05,263 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,263 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,264 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,264 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,265 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,265 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:05,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:05,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:05,266 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:05,266 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,266 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:05,266 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:05,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:05,267 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,267 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:05,267 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:05,267 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:05,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:05,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:05,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:05,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,268 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,268 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,269 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:05,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18b4ea21[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:05,270 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:05,270 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,271 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:05,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:05,271 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,271 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:05,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:05,273 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,273 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:05,273 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,273 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:05,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:05,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:05,275 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:05,275 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:05,275 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:05,275 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:05,275 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:05,278 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,278 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,278 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,279 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,280 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,282 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,282 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,282 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:05,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:05,282 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,282 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,282 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,283 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,283 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:05,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:05,283 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:05,283 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,283 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,383 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:05,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:05,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:05,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:05,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:05,386 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:05,387 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,387 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:05,387 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:05,387 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,387 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,388 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,388 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,388 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,388 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:05,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:05,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:05,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:05,390 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,390 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:05,390 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:05,390 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:05,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:05,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,392 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,392 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,392 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:05,394 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:05,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d89a755[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:05,394 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,400 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,400 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:05,400 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,400 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:05,400 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:05,400 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:05,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:05,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:05,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:05,404 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:05,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:05,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:05,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:05,404 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:05,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,406 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,407 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,409 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,409 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:05,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:05,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:05,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,410 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:05,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,410 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:05,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:05,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229365452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/f81f9c80db7b4bd7a37d467532d595ef 2023-04-23 05:55:05,511 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:05,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:05,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:05,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:05,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:05,514 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:05,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:05,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:05,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,516 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,516 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:05,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:05,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:05,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:05,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:05,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:05,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:05,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:05,518 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,518 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:05,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:05,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:05,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:05,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:05,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:05,520 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,520 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:05,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c4ae1ae[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:05,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:05,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,523 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:05,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:05,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:05,525 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:05,525 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,525 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,525 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:05,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:05,528 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:05,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:05,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:05,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:05,528 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:05,530 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,530 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,530 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,531 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:05,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229365532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,535 DEBUG [Thread-1545] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:55:05,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,535 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,536 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:05,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:05,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,536 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:05,536 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:05,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:05,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229365579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,581 DEBUG [Thread-1543] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:55:05,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:05,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52816 deadline: 1682229365582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,584 DEBUG [Thread-1541] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:55:05,636 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:05,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:05,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:05,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:05,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:05,640 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:05,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:05,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:05,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,641 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,642 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,642 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:05,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:05,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:05,643 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:05,643 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,643 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:05,643 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:05,643 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:05,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:05,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:05,644 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:05,645 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,645 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,645 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,645 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:05,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:05,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61230011[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:05,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:05,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,649 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:05,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:05,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:05,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:05,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:05,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:05,654 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:05,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:05,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:05,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:05,654 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:05,655 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,655 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,655 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,656 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,657 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:05,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:05,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:05,659 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,660 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:05,660 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:05,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:05,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229365754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,760 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:05,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:05,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:05,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:05,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:05,763 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:05,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:05,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:05,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,767 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,767 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:05,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:05,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:05,768 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,768 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:05,768 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:05,768 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:05,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:05,770 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,770 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,771 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:05,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:05,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef90bf6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:05,772 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:05,772 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:05,774 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:05,774 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:05,774 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,774 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:05,778 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:05,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:05,780 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:05,780 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:05,781 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:05,781 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:05,781 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:05,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,784 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,786 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,786 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,786 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,786 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:05,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:05,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:05,786 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,787 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,787 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,787 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,787 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,787 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:05,787 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,787 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:05,887 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:05,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:05,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:05,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:05,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:05,890 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:05,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:05,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:05,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:05,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:05,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:05,894 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,894 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:05,894 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:05,894 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:05,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:05,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,897 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,897 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,897 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:05,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:05,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8abc73[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:05,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:05,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,900 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,900 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:05,900 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:05,900 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:05,900 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,900 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:05,901 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:05,901 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,901 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:05,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:05,904 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:05,904 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:05,904 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:05,904 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:05,904 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:05,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:05,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:05,906 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,907 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:05,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:05,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:05,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:05,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:05,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:05,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:05,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:05,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:05,910 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:05,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:05,911 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:05,911 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:05,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:05,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/9a668450096f482bb33a114afb4e641a 2023-04-23 05:55:06,011 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:06,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:06,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:06,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:06,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:06,015 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:06,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:06,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:06,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,016 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,017 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:06,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:06,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:06,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:06,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:06,018 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:06,018 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,018 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:06,019 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:06,019 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:06,020 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,020 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:06,020 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:06,020 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:06,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:06,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:06,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:06,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,022 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:06,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a6799d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:06,024 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:06,024 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,025 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:06,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:06,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:06,025 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,025 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:06,027 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:06,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,027 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:06,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:06,030 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:06,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:06,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:06,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:06,030 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:06,032 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,032 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:06,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,032 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,033 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:06,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:06,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:06,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:06,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,036 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:06,037 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:06,037 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:06,037 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:06,037 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:06,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:06,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:06,137 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:06,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:06,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:06,140 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:06,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:06,141 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:06,141 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,141 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,142 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:06,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:06,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:06,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,143 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:06,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:06,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:06,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:06,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:06,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:06,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:06,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:06,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,147 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,148 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:06,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cfcccb0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:06,149 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:06,149 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,150 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,150 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:06,150 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:06,150 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:06,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:06,152 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,152 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:06,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:06,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:06,154 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:06,154 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:06,154 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:06,154 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:06,154 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:06,155 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,156 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:06,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,156 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:06,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:06,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:06,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:06,163 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:06,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:06,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:06,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,164 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:06,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:06,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:06,164 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:06,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:06,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:06,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229366257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,264 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:06,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:06,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:06,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:06,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:06,267 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:06,267 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,267 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:06,267 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:06,268 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,268 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,269 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:06,269 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:06,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,270 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,270 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:06,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:06,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:06,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:06,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:06,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:06,271 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,272 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:06,272 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:06,272 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:06,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,274 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:06,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:06,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1accc062[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:06,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:06,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,277 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:06,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:06,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:06,279 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,279 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,279 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:06,280 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:06,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:06,282 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:06,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:06,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:06,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:06,282 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:06,284 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,284 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,284 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:06,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:06,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:06,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:06,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:06,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:06,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:06,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:06,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:06,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:06,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:06,292 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:06,292 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:06,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=753 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/eae45b335b83451ebe03783a81197d35 2023-04-23 05:55:06,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/f81f9c80db7b4bd7a37d467532d595ef as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f81f9c80db7b4bd7a37d467532d595ef 2023-04-23 05:55:06,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f81f9c80db7b4bd7a37d467532d595ef, entries=150, sequenceid=753, filesize=11.9 K 2023-04-23 05:55:06,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/9a668450096f482bb33a114afb4e641a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9a668450096f482bb33a114afb4e641a 2023-04-23 05:55:06,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9a668450096f482bb33a114afb4e641a, entries=150, sequenceid=753, filesize=11.9 K 2023-04-23 05:55:06,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/eae45b335b83451ebe03783a81197d35 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/eae45b335b83451ebe03783a81197d35 2023-04-23 05:55:06,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/eae45b335b83451ebe03783a81197d35, entries=150, sequenceid=753, filesize=11.9 K 2023-04-23 05:55:06,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 911c079a878d9ede8370cba6d74e3efd in 1250ms, sequenceid=753, compaction requested=true 2023-04-23 05:55:06,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:06,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:06,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:06,340 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:06,340 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:06,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:06,342 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:06,342 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/A is initiating minor compaction (all files) 2023-04-23 05:55:06,343 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/A in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,343 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:06,343 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/a704432cb4b5443b8a7d91fcff0d394b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/dde6f51a6a874435beb7896f2fb9527e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f81f9c80db7b4bd7a37d467532d595ef] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=36.2 K 2023-04-23 05:55:06,343 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/B is initiating minor compaction (all files) 2023-04-23 05:55:06,343 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/B in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,343 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/8683167810e04f849fd7cd492a500793, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/3e5e9cb7e9a84cfaa003c9cfeb805d87, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9a668450096f482bb33a114afb4e641a] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=36.2 K 2023-04-23 05:55:06,343 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a704432cb4b5443b8a7d91fcff0d394b, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1682229302492 2023-04-23 05:55:06,343 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 8683167810e04f849fd7cd492a500793, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1682229302492 2023-04-23 05:55:06,344 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting dde6f51a6a874435beb7896f2fb9527e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1682229302875 2023-04-23 05:55:06,344 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3e5e9cb7e9a84cfaa003c9cfeb805d87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1682229302875 2023-04-23 05:55:06,344 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting f81f9c80db7b4bd7a37d467532d595ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682229303966 2023-04-23 05:55:06,344 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9a668450096f482bb33a114afb4e641a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682229303966 2023-04-23 05:55:06,351 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#A#compaction#242 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:06,351 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#B#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:06,360 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/cfc511f9b7d349979686e3c010749e86 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cfc511f9b7d349979686e3c010749e86 2023-04-23 05:55:06,360 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/82b41b740d0b46088ae872dc3cf40299 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/82b41b740d0b46088ae872dc3cf40299 2023-04-23 05:55:06,365 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/A of 911c079a878d9ede8370cba6d74e3efd into cfc511f9b7d349979686e3c010749e86(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:06,365 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:06,365 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/A, priority=13, startTime=1682229306340; duration=0sec 2023-04-23 05:55:06,365 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:06,365 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:06,366 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/B of 911c079a878d9ede8370cba6d74e3efd into 82b41b740d0b46088ae872dc3cf40299(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:06,366 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:06,366 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/B, priority=13, startTime=1682229306340; duration=0sec 2023-04-23 05:55:06,366 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:06,366 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:06,366 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/C is initiating minor compaction (all files) 2023-04-23 05:55:06,367 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/C in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,367 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/410dfa6905934fafb240c5adfaf70203, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a5e72b1c54b44567acab6eb1d27b7140, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/eae45b335b83451ebe03783a81197d35] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=36.2 K 2023-04-23 05:55:06,367 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 410dfa6905934fafb240c5adfaf70203, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1682229302492 2023-04-23 05:55:06,367 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a5e72b1c54b44567acab6eb1d27b7140, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1682229302875 2023-04-23 05:55:06,368 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting eae45b335b83451ebe03783a81197d35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682229303966 2023-04-23 05:55:06,374 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#C#compaction#244 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:06,385 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/0a17a6f7d58f47309acb2e6a97133597 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0a17a6f7d58f47309acb2e6a97133597 2023-04-23 05:55:06,391 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/C of 911c079a878d9ede8370cba6d74e3efd into 0a17a6f7d58f47309acb2e6a97133597(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:06,392 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:06,392 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/C, priority=13, startTime=1682229306340; duration=0sec 2023-04-23 05:55:06,392 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:06,392 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:06,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:06,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:06,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:06,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:06,395 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:06,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:06,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:06,396 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,396 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:06,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:06,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:06,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:06,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:06,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:06,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:06,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:06,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:06,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:06,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:06,399 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,399 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:06,400 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:55:06,400 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:06,400 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:06,400 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:06,400 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:06,400 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:06,400 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:06,808 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/581c92003eb0441fa375744cb713edb5 2023-04-23 05:55:06,818 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/7c0ad94e3ad34ad69d96add734725fa3 2023-04-23 05:55:06,828 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=776 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/48a7d4352d8b43d8ad3ae684325882ba 2023-04-23 05:55:06,833 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/581c92003eb0441fa375744cb713edb5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/581c92003eb0441fa375744cb713edb5 2023-04-23 05:55:06,837 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/581c92003eb0441fa375744cb713edb5, entries=150, sequenceid=776, filesize=11.9 K 2023-04-23 05:55:06,838 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/7c0ad94e3ad34ad69d96add734725fa3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7c0ad94e3ad34ad69d96add734725fa3 2023-04-23 05:55:06,842 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7c0ad94e3ad34ad69d96add734725fa3, entries=150, sequenceid=776, filesize=11.9 K 2023-04-23 05:55:06,843 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/48a7d4352d8b43d8ad3ae684325882ba as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/48a7d4352d8b43d8ad3ae684325882ba 2023-04-23 05:55:06,847 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/48a7d4352d8b43d8ad3ae684325882ba, entries=150, sequenceid=776, filesize=11.9 K 2023-04-23 05:55:06,848 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 911c079a878d9ede8370cba6d74e3efd in 448ms, sequenceid=776, compaction requested=false 2023-04-23 05:55:06,848 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:06,848 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,848 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:06,848 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:06,848 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:06,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:06,853 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:06,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@516e8e6c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:06,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:06,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,858 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:06,858 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:06,858 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:06,858 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,858 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:06,859 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:06,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:06,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:06,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:06,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:06,863 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:06,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:06,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:06,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:06,863 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:06,865 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,865 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,865 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:06,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:06,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:06,866 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,867 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:06,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:06,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:06,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:06,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:06,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:06,873 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:06,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,872 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:06,873 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,873 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:06,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:06,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:06,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:06,973 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:06,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:06,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:06,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:06,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:06,977 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:06,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:06,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:06,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,979 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:06,979 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:06,979 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,979 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:06,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:06,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:06,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:06,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:06,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:06,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:06,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:06,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:06,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:06,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:06,981 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,981 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:06,982 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:06,982 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:06,982 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:06,982 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:06,982 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,982 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:06,982 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:06,983 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,984 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:06,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:06,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c26f4d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:06,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:06,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:06,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:06,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:06,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:06,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:06,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:06,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:06,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:06,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:06,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:06,992 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:06,992 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:06,992 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:06,992 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:06,992 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:06,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:06,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:06,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:06,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:06,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:06,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:06,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:06,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:06,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:06,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:07,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:07,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,000 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:07,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,001 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:07,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:07,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:07,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:07,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:07,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:07,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:07,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:07,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:07,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:07,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/b2a654cc565944f98a3f6d64a7d086c8 2023-04-23 05:55:07,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229307073, max timestamp=1682229307087Num uniques 859; 2023-04-23 05:55:07,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:55:07,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229307073, max timestamp=1682229307087Num uniques 859; 2023-04-23 05:55:07,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:55:07,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229307073, max timestamp=1682229307087Num uniques 859; 2023-04-23 05:55:07,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:55:07,101 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:07,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:07,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:07,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:07,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:07,104 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:07,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:07,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:07,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,106 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:07,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:07,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:07,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:07,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:07,108 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,108 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:07,108 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:07,108 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:07,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:07,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,110 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:07,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:07,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d31b849[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:07,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:07,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:07,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:07,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:07,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:07,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:07,115 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:07,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:07,117 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:07,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:07,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:07,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:07,117 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:07,120 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,120 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,120 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:07,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:07,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:07,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,125 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:07,125 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:07,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:07,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229367137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,225 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:07,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:07,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:07,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:07,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:07,228 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:07,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:07,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:07,229 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,229 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:07,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:07,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:07,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:07,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:07,233 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,233 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:07,233 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:07,233 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:07,235 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:07,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@182a85d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:07,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:07,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,239 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:07,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:07,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:07,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:07,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:07,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:07,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229367240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:07,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:07,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:07,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:07,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:07,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:07,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:07,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,247 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,250 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:07,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:07,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,250 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:07,251 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:07,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:07,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229367261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,351 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:07,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:07,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:07,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:07,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:07,354 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:07,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:07,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:07,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,357 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:07,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:07,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:07,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:07,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:07,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:07,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:07,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:07,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:07,360 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:07,362 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:07,362 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@243a5bef[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:07,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,363 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:07,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:07,363 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:07,363 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:07,365 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,365 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:07,366 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,366 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:07,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:07,367 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:07,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:07,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:07,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:07,368 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:07,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,374 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,374 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,374 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:07,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:07,375 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,375 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,375 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,375 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,375 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:07,375 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,375 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,375 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:07,375 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:07,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229367444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/fadc03b7b64b49e7941aaa8286a9b3ba 2023-04-23 05:55:07,475 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:07,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:07,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:07,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:07,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:07,479 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:07,479 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,479 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:07,479 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:07,480 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,480 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,484 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,484 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,484 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:07,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:07,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:07,486 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:07,486 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,486 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:07,486 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:07,486 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:07,486 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,486 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:07,487 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:07,487 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:07,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:07,489 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,489 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:07,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2086f022[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:07,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:07,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,492 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:07,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:07,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:07,492 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,492 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,494 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:07,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:07,496 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:07,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:07,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:07,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:07,496 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:07,498 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,498 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,498 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,503 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,503 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,503 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,503 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:07,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:07,503 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:07,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,504 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,504 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,504 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,504 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:07,504 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:07,504 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,620 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:07,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:07,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:07,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:07,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:07,623 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:07,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:07,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:07,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:07,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:07,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:07,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:07,626 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:07,626 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:07,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:07,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:07,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:07,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:07,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5085e78b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:07,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:07,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,631 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:07,632 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,632 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:07,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:07,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:07,634 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:07,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:07,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:07,636 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:07,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:07,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:07,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:07,637 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:07,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,639 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,642 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:07,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:07,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:07,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,643 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:07,643 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:07,743 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:07,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:07,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:07,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:07,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:07,747 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:07,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:07,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:07,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:07,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229367748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:07,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:07,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:07,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:07,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:07,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:07,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:07,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,754 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:07,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:07,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44516d86[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:07,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:07,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,757 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:07,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:07,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:07,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:07,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:07,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:07,762 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:07,762 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:07,762 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:07,762 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:07,762 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:07,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,765 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,769 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,769 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:07,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:07,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:07,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,769 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:07,769 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,870 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:07,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:07,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:07,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:07,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:07,873 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:07,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:07,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:07,874 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,874 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=787 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/725d83ae159647de92bb45ac770212d7 2023-04-23 05:55:07,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:07,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:07,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:07,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:07,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:07,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:07,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/b2a654cc565944f98a3f6d64a7d086c8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2a654cc565944f98a3f6d64a7d086c8 2023-04-23 05:55:07,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2a654cc565944f98a3f6d64a7d086c8, entries=150, sequenceid=787, filesize=11.9 K 2023-04-23 05:55:07,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:07,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:07,884 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,884 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:07,884 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:07,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/fadc03b7b64b49e7941aaa8286a9b3ba as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fadc03b7b64b49e7941aaa8286a9b3ba 2023-04-23 05:55:07,884 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:07,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fadc03b7b64b49e7941aaa8286a9b3ba, entries=150, sequenceid=787, filesize=11.9 K 2023-04-23 05:55:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,890 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:07,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/725d83ae159647de92bb45ac770212d7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/725d83ae159647de92bb45ac770212d7 2023-04-23 05:55:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:07,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c2a3c0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:07,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:07,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,894 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:07,894 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:07,895 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/725d83ae159647de92bb45ac770212d7, entries=150, sequenceid=787, filesize=11.9 K 2023-04-23 05:55:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:07,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:07,896 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:07,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,896 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,896 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:07,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:07,899 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:07,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:07,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:07,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:07,899 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:07,899 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 911c079a878d9ede8370cba6d74e3efd in 860ms, sequenceid=787, compaction requested=true 2023-04-23 05:55:07,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:07,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:07,900 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:07,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:07,900 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:07,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:07,900 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,900 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,901 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:07,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,901 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:07,901 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/A is initiating minor compaction (all files) 2023-04-23 05:55:07,901 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/A in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,902 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cfc511f9b7d349979686e3c010749e86, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/581c92003eb0441fa375744cb713edb5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2a654cc565944f98a3f6d64a7d086c8] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=36.3 K 2023-04-23 05:55:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,902 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:07,902 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/B is initiating minor compaction (all files) 2023-04-23 05:55:07,902 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/B in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:07,902 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/82b41b740d0b46088ae872dc3cf40299, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7c0ad94e3ad34ad69d96add734725fa3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fadc03b7b64b49e7941aaa8286a9b3ba] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=36.3 K 2023-04-23 05:55:07,903 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting cfc511f9b7d349979686e3c010749e86, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682229303966 2023-04-23 05:55:07,903 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 82b41b740d0b46088ae872dc3cf40299, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682229303966 2023-04-23 05:55:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:07,903 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,903 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 581c92003eb0441fa375744cb713edb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1682229305136 2023-04-23 05:55:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,903 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7c0ad94e3ad34ad69d96add734725fa3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1682229305136 2023-04-23 05:55:07,903 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b2a654cc565944f98a3f6d64a7d086c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1682229307035 2023-04-23 05:55:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:07,904 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting fadc03b7b64b49e7941aaa8286a9b3ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1682229307035 2023-04-23 05:55:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,905 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:07,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:07,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:07,909 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:07,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:07,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:07,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:07,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:07,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:07,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:07,909 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:07,910 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:07,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:07,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:07,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:07,916 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#A#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:07,916 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#B#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:08,010 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:08,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:08,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:08,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:08,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:08,013 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:08,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,014 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:08,014 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:08,020 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,020 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:08,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:08,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:08,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:08,025 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-23 05:55:08,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:08,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:08,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:08,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:08,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:08,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:08,041 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=823 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/c73e36b22ff64e01b1c59e08bf043023 2023-04-23 05:55:08,052 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=823 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/61496f1c8205478fb3e74102cd9196e6 2023-04-23 05:55:08,064 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=823 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/ca41f593090c4a2bbe0c4ced4c202b49 2023-04-23 05:55:08,068 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/c73e36b22ff64e01b1c59e08bf043023 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c73e36b22ff64e01b1c59e08bf043023 2023-04-23 05:55:08,073 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c73e36b22ff64e01b1c59e08bf043023, entries=150, sequenceid=823, filesize=11.9 K 2023-04-23 05:55:08,074 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/61496f1c8205478fb3e74102cd9196e6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61496f1c8205478fb3e74102cd9196e6 2023-04-23 05:55:08,077 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61496f1c8205478fb3e74102cd9196e6, entries=150, sequenceid=823, filesize=11.9 K 2023-04-23 05:55:08,078 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/ca41f593090c4a2bbe0c4ced4c202b49 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca41f593090c4a2bbe0c4ced4c202b49 2023-04-23 05:55:08,096 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca41f593090c4a2bbe0c4ced4c202b49, entries=150, sequenceid=823, filesize=11.9 K 2023-04-23 05:55:08,097 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 911c079a878d9ede8370cba6d74e3efd in 72ms, sequenceid=823, compaction requested=true 2023-04-23 05:55:08,097 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:08,097 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:08,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:08,099 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,099 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,099 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,099 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:08,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2552425a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:08,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:08,102 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,103 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:08,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:08,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:08,103 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,103 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:08,104 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:08,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:08,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:08,110 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:08,110 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:08,110 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:08,110 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:08,110 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:08,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,115 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,115 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,120 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,120 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:08,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:08,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:08,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,121 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:08,121 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:08,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,221 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:08,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:08,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:08,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:08,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:08,225 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:08,225 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,225 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:08,225 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:08,225 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,225 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,226 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,227 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,227 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,227 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:08,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:08,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:08,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:08,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:08,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:08,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:08,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:08,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:08,231 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,231 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:08,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:08,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:08,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:08,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:08,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:08,233 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,234 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,234 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:08,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a9450c4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:08,235 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:08,236 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,237 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,237 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:08,237 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:08,237 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:08,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:08,239 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,239 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:08,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:08,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:08,241 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:08,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:08,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:08,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:08,241 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:08,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:08,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:08,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,247 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,248 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:08,248 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,248 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:08,248 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,248 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,248 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:08,266 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:08,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:08,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:08,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:08,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:08,267 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:08,267 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:08,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229308315, max timestamp=1682229308330Num uniques 859; 2023-04-23 05:55:08,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:08,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229308315, max timestamp=1682229308330Num uniques 859; 2023-04-23 05:55:08,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:08,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229308315, max timestamp=1682229308330Num uniques 859; 2023-04-23 05:55:08,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:08,340 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/f6cc92c297be4f7386f62597705e66be as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f6cc92c297be4f7386f62597705e66be 2023-04-23 05:55:08,340 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/75f0f2753e0043e29a2a488681b79d31 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/75f0f2753e0043e29a2a488681b79d31 2023-04-23 05:55:08,348 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/A of 911c079a878d9ede8370cba6d74e3efd into f6cc92c297be4f7386f62597705e66be(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:08,348 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/B of 911c079a878d9ede8370cba6d74e3efd into 75f0f2753e0043e29a2a488681b79d31(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:08,348 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:08,348 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/B, priority=13, startTime=1682229307900; duration=0sec 2023-04-23 05:55:08,348 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:08,348 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:08,348 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:08,348 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:08,349 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/A, priority=13, startTime=1682229307900; duration=0sec 2023-04-23 05:55:08,349 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:08,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:08,350 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:08,350 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/C is initiating minor compaction (all files) 2023-04-23 05:55:08,350 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/C in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,350 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0a17a6f7d58f47309acb2e6a97133597, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/48a7d4352d8b43d8ad3ae684325882ba, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/725d83ae159647de92bb45ac770212d7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca41f593090c4a2bbe0c4ced4c202b49] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=48.2 K 2023-04-23 05:55:08,351 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 0a17a6f7d58f47309acb2e6a97133597, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=753, earliestPutTs=1682229303966 2023-04-23 05:55:08,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:08,351 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 48a7d4352d8b43d8ad3ae684325882ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=776, earliestPutTs=1682229305136 2023-04-23 05:55:08,352 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 725d83ae159647de92bb45ac770212d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1682229307035 2023-04-23 05:55:08,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:08,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:08,352 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:08,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,352 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ca41f593090c4a2bbe0c4ced4c202b49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=823, earliestPutTs=1682229307125 2023-04-23 05:55:08,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:08,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:08,353 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,353 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:08,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:08,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:08,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:08,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:08,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:08,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:08,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:08,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:08,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:08,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:08,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:08,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:08,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:08,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:08,360 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#C#compaction#257 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:08,360 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:08,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15662b4d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:08,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:08,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:08,364 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:08,364 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,364 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:08,369 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,369 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:08,369 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,369 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:08,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:08,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:08,371 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:08,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:08,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:08,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:08,372 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:08,373 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,373 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,373 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,373 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,375 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,377 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:08,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:08,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,377 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:08,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,378 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,378 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:08,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:08,378 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,378 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,378 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:08,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229368402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,478 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:08,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:08,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:08,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:08,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:08,481 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:08,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:08,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:08,482 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,482 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,483 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:08,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:08,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:08,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:08,485 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,485 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:08,485 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:08,485 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:08,485 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:08,487 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:08,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329d5682[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:08,489 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:08,489 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,490 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,491 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:08,491 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:08,491 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:08,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:08,492 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:08,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:08,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:08,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:08,495 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:08,495 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:08,495 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:08,495 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:08,495 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:08,496 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,496 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,496 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,499 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,501 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,501 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,501 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,502 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,502 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,501 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:08,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:08,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:08,502 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,502 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,502 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:08,502 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,502 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,502 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:08,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229368506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,602 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:08,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:08,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:08,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:08,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:08,605 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:08,605 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,605 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:08,605 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:08,606 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,606 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,607 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:08,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:08,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:08,609 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:08,609 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:08,611 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,611 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,612 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,612 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:08,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:08,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea8219[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:08,613 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:08,613 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:08,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:08,614 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,614 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:08,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:08,616 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,617 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,617 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,617 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:08,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:08,619 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:08,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:08,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:08,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:08,620 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:08,622 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,622 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,622 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,627 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,627 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,627 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,627 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:08,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:08,627 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,627 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:08,627 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,628 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:08,628 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:08,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,627 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/605913b5f2ad4ca78564aa39de89a1f1 2023-04-23 05:55:08,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:08,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229368709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,728 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:08,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:08,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:08,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:08,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:08,732 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:08,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:08,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:08,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,733 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,733 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,733 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:08,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:08,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:08,735 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,735 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:08,735 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:08,735 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:08,735 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:08,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:08,738 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,738 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,738 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,738 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:08,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:08,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d80e010[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:08,740 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,741 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:08,741 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,741 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:08,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:08,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:08,742 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:08,742 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,742 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,742 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:08,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:08,745 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:08,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:08,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:08,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:08,745 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:08,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:08,759 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:08,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:08,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,760 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,760 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:08,769 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/79b8d45be0924d0e900044667de56b33 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/79b8d45be0924d0e900044667de56b33 2023-04-23 05:55:08,774 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/C of 911c079a878d9ede8370cba6d74e3efd into 79b8d45be0924d0e900044667de56b33(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:08,774 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:08,774 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/C, priority=12, startTime=1682229307900; duration=0sec 2023-04-23 05:55:08,774 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:08,860 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:08,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:08,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:08,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:08,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:08,867 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:08,867 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,867 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:08,867 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:08,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,869 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:08,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:08,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:08,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:08,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:08,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:08,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:08,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:08,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:08,871 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,871 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:08,871 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:08,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:08,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:08,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:08,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:08,873 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:08,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@599890ae[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:08,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:08,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:08,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:08,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:08,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:08,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:08,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:08,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:08,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:08,883 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:08,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:08,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:08,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:08,883 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:08,884 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,884 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,885 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,886 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,886 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:08,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,889 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:08,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:08,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:08,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:08,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:08,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,889 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:08,890 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:08,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:08,990 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:08,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:08,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:08,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:08,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:08,993 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:08,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:08,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:08,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:08,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:08,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:08,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:08,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:08,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:08,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:08,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:08,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:08,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:08,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:08,996 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,996 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:08,997 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:08,997 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:08,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:08,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:08,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:08,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:08,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,999 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:08,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:08,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:09,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:09,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169f459[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:09,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:09,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:09,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:09,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:09,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:09,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:09,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:09,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:09,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:09,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:09,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:09,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:09,007 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:09,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:09,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:09,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:09,007 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:09,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:09,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:09,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:09,010 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:09,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:09,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:09,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:09,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:09,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:09,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:09,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:09,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:09,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:09,013 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:09,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:09,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:09,013 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:09,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:09,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229369013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/bea8237781f34c7ba6620a29e176820e 2023-04-23 05:55:09,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=834 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/f7c881349e054ae78d369f8916691b68 2023-04-23 05:55:09,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/605913b5f2ad4ca78564aa39de89a1f1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/605913b5f2ad4ca78564aa39de89a1f1 2023-04-23 05:55:09,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/605913b5f2ad4ca78564aa39de89a1f1, entries=200, sequenceid=834, filesize=14.3 K 2023-04-23 05:55:09,105 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/bea8237781f34c7ba6620a29e176820e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/bea8237781f34c7ba6620a29e176820e 2023-04-23 05:55:09,108 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/bea8237781f34c7ba6620a29e176820e, entries=150, sequenceid=834, filesize=11.9 K 2023-04-23 05:55:09,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/f7c881349e054ae78d369f8916691b68 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/f7c881349e054ae78d369f8916691b68 2023-04-23 05:55:09,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/f7c881349e054ae78d369f8916691b68, entries=150, sequenceid=834, filesize=11.9 K 2023-04-23 05:55:09,113 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 911c079a878d9ede8370cba6d74e3efd in 847ms, sequenceid=834, compaction requested=true 2023-04-23 05:55:09,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:09,113 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:09,114 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:09,114 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:09,114 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:09,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:09,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:09,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:09,115 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:09,115 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/B is initiating minor compaction (all files) 2023-04-23 05:55:09,115 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:09,115 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/B in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:09,115 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/A is initiating minor compaction (all files) 2023-04-23 05:55:09,115 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/75f0f2753e0043e29a2a488681b79d31, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61496f1c8205478fb3e74102cd9196e6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/bea8237781f34c7ba6620a29e176820e] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=36.4 K 2023-04-23 05:55:09,115 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/A in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:09,115 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f6cc92c297be4f7386f62597705e66be, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c73e36b22ff64e01b1c59e08bf043023, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/605913b5f2ad4ca78564aa39de89a1f1] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=38.8 K 2023-04-23 05:55:09,115 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 75f0f2753e0043e29a2a488681b79d31, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1682229307035 2023-04-23 05:55:09,115 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting f6cc92c297be4f7386f62597705e66be, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=787, earliestPutTs=1682229307035 2023-04-23 05:55:09,116 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 61496f1c8205478fb3e74102cd9196e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=823, earliestPutTs=1682229307125 2023-04-23 05:55:09,116 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c73e36b22ff64e01b1c59e08bf043023, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=823, earliestPutTs=1682229307125 2023-04-23 05:55:09,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:09,117 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting bea8237781f34c7ba6620a29e176820e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1682229308260 2023-04-23 05:55:09,117 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 605913b5f2ad4ca78564aa39de89a1f1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1682229308252 2023-04-23 05:55:09,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:09,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:09,118 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:09,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:09,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:09,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:09,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:09,120 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:09,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:09,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:09,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:09,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:09,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:09,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:09,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:09,121 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:09,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:09,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:09,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:09,122 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:09,122 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:09,122 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:09,122 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-23 05:55:09,122 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:09,122 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:09,122 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:09,123 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:09,123 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:09,123 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:09,124 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#B#compaction#260 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:09,125 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#A#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:09,138 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=874 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/9d230443fc844d0eab78ab52f701d6f1 2023-04-23 05:55:09,138 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/b4f83cdb84424e929b737d9e47cfb374 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/b4f83cdb84424e929b737d9e47cfb374 2023-04-23 05:55:09,141 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/3ee85ea75d5144aaa61cebeb2585efd3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3ee85ea75d5144aaa61cebeb2585efd3 2023-04-23 05:55:09,144 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/B of 911c079a878d9ede8370cba6d74e3efd into b4f83cdb84424e929b737d9e47cfb374(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:09,144 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:09,144 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/B, priority=13, startTime=1682229309114; duration=0sec 2023-04-23 05:55:09,145 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:09,145 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-23 05:55:09,146 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-23 05:55:09,146 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-23 05:55:09,146 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. because compaction request was cancelled 2023-04-23 05:55:09,147 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/A of 911c079a878d9ede8370cba6d74e3efd into 3ee85ea75d5144aaa61cebeb2585efd3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:09,147 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:09,148 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/A, priority=13, startTime=1682229309113; duration=0sec 2023-04-23 05:55:09,148 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:09,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:09,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:09,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:09,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229369286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:09,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229369388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:09,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229369515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,552 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=874 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/5e461e1d272b4dcea32e0ef0c8d9bb6f 2023-04-23 05:55:09,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:09,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52824 deadline: 1682229369562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,565 DEBUG [Thread-1545] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:55:09,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:09,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229369591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:09,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52816 deadline: 1682229369603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,607 DEBUG [Thread-1541] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:55:09,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:09,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52786 deadline: 1682229369607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,611 DEBUG [Thread-1543] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:55:09,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:09,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229369895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,962 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=874 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/d1d3ec1ca6ca479ebe48a4eeaa09b23a 2023-04-23 05:55:09,967 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/9d230443fc844d0eab78ab52f701d6f1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/9d230443fc844d0eab78ab52f701d6f1 2023-04-23 05:55:09,971 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/9d230443fc844d0eab78ab52f701d6f1, entries=150, sequenceid=874, filesize=11.9 K 2023-04-23 05:55:09,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/5e461e1d272b4dcea32e0ef0c8d9bb6f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/5e461e1d272b4dcea32e0ef0c8d9bb6f 2023-04-23 05:55:09,978 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/5e461e1d272b4dcea32e0ef0c8d9bb6f, entries=150, sequenceid=874, filesize=11.9 K 2023-04-23 05:55:09,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/d1d3ec1ca6ca479ebe48a4eeaa09b23a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/d1d3ec1ca6ca479ebe48a4eeaa09b23a 2023-04-23 05:55:09,982 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/d1d3ec1ca6ca479ebe48a4eeaa09b23a, entries=150, sequenceid=874, filesize=11.9 K 2023-04-23 05:55:09,983 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 911c079a878d9ede8370cba6d74e3efd in 861ms, sequenceid=874, compaction requested=true 2023-04-23 05:55:09,983 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:09,983 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:09,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:09,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:09,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:09,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:09,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50c25427[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:09,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:09,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:09,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:09,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:09,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:09,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:09,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:09,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:09,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:09,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:09,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:09,994 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:09,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:09,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:09,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:09,994 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:09,995 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,995 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:09,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,995 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:09,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:09,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:09,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:10,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:10,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:10,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:10,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:10,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:10,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:10,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:10,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:10,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:10,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:10,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:10,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:10,003 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:10,003 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:10,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:10,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:10,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:10,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:10,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:10,103 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:10,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:10,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:10,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:10,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:10,106 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:10,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:10,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:10,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:10,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:10,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:10,109 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:10,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:10,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:10,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:10,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:10,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:10,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:10,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:10,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:10,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:10,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:10,111 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:10,111 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:10,111 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:10,111 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:10,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:10,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:10,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:10,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:10,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:10,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:10,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:10,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229310435, max timestamp=1682229310462Num uniques 859; 2023-04-23 05:55:10,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229310435, max timestamp=1682229310462Num uniques 859; 2023-04-23 05:55:10,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:10,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:10,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229310435, max timestamp=1682229310462Num uniques 859; 2023-04-23 05:55:10,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:10,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:10,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229370513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:10,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:10,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229370519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:10,523 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=886 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/ba2cbccf7ef94b0bb7094da573c45ba5 2023-04-23 05:55:10,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:10,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229370616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:10,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:10,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229370819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:10,934 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=886 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/55b9d7529b7346269d0a912ba4859573 2023-04-23 05:55:11,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:11,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229371123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,347 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=886 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/9306f390719244b2afe2732e8208b305 2023-04-23 05:55:11,351 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/ba2cbccf7ef94b0bb7094da573c45ba5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ba2cbccf7ef94b0bb7094da573c45ba5 2023-04-23 05:55:11,355 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ba2cbccf7ef94b0bb7094da573c45ba5, entries=150, sequenceid=886, filesize=11.9 K 2023-04-23 05:55:11,355 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/55b9d7529b7346269d0a912ba4859573 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/55b9d7529b7346269d0a912ba4859573 2023-04-23 05:55:11,359 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/55b9d7529b7346269d0a912ba4859573, entries=150, sequenceid=886, filesize=11.9 K 2023-04-23 05:55:11,360 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/9306f390719244b2afe2732e8208b305 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9306f390719244b2afe2732e8208b305 2023-04-23 05:55:11,363 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9306f390719244b2afe2732e8208b305, entries=150, sequenceid=886, filesize=11.9 K 2023-04-23 05:55:11,366 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 911c079a878d9ede8370cba6d74e3efd in 1255ms, sequenceid=886, compaction requested=true 2023-04-23 05:55:11,366 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:11,366 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:11,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:11,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:11,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:11,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:11,368 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:11,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:11,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a38ae4f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:11,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:11,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:11,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:11,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:11,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:11,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:11,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:11,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:11,374 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,374 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:11,374 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:11,374 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:11,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:11,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:11,376 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:11,376 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:11,376 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:11,376 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:11,376 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:11,378 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:11,378 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:11,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:11,378 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:11,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:11,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:11,380 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:11,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:11,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,382 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:11,382 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:11,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:11,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:11,382 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:11,382 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:11,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:11,382 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:11,382 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:11,382 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:11,382 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:11,383 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:11,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:11,383 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:11,383 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:11,383 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:11,483 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:11,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:11,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:11,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:11,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:11,486 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:11,486 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:11,486 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:11,486 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:11,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:11,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:11,488 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:11,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:11,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:11,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:11,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:11,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:11,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:11,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:11,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:11,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:11,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:11,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:11,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:11,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:11,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:11,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:11,492 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-23 05:55:11,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:11,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:11,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:11,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:11,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:11,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:11,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:11,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:11,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:11,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229371651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:11,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229371755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:11,902 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/c9c54bbca2b64cb9bb9488a0e4cdee5a 2023-04-23 05:55:11,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:11,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229371957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:12,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229372262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,313 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/cfebe44c4ede4a9abefdda3ec2b255ca 2023-04-23 05:55:12,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:12,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52830 deadline: 1682229372525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,527 DEBUG [Thread-1547] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4124 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:55:12,723 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/a4930698f49d4c98a8a8f419e7cd73e4 2023-04-23 05:55:12,727 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/c9c54bbca2b64cb9bb9488a0e4cdee5a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c9c54bbca2b64cb9bb9488a0e4cdee5a 2023-04-23 05:55:12,731 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c9c54bbca2b64cb9bb9488a0e4cdee5a, entries=150, sequenceid=923, filesize=11.9 K 2023-04-23 05:55:12,732 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/cfebe44c4ede4a9abefdda3ec2b255ca as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/cfebe44c4ede4a9abefdda3ec2b255ca 2023-04-23 05:55:12,739 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/cfebe44c4ede4a9abefdda3ec2b255ca, entries=150, sequenceid=923, filesize=11.9 K 2023-04-23 05:55:12,740 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/a4930698f49d4c98a8a8f419e7cd73e4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a4930698f49d4c98a8a8f419e7cd73e4 2023-04-23 05:55:12,743 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a4930698f49d4c98a8a8f419e7cd73e4, entries=150, sequenceid=923, filesize=11.9 K 2023-04-23 05:55:12,745 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 911c079a878d9ede8370cba6d74e3efd in 1253ms, sequenceid=923, compaction requested=true 2023-04-23 05:55:12,745 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:12,745 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:12,745 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:12,745 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:12,745 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:12,749 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:12,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,752 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:12,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:12,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:12,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:12,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37b84646[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:12,754 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:12,754 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,755 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:12,755 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:12,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:12,757 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:12,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:12,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:12,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:12,760 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:12,760 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:12,760 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:12,760 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:12,760 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:12,761 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,761 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,761 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:12,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,762 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:12,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:12,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:12,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:12,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:12,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:12,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:12,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:12,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:12,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,766 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:12,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:12,767 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:12,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:12,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:12,767 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:55:12,767 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:12,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:12,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:12,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:12,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:12,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:12,775 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=934 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/edb2ec046eed4572bf0f54910f248f33 2023-04-23 05:55:12,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=934 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/61a97f406f2b42cd83baf1ae9e9f63ed 2023-04-23 05:55:12,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=934 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/7a5989d95fe24349b394c3cae2a2f279 2023-04-23 05:55:12,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/edb2ec046eed4572bf0f54910f248f33 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/edb2ec046eed4572bf0f54910f248f33 2023-04-23 05:55:12,803 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/edb2ec046eed4572bf0f54910f248f33, entries=150, sequenceid=934, filesize=11.9 K 2023-04-23 05:55:12,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/61a97f406f2b42cd83baf1ae9e9f63ed as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61a97f406f2b42cd83baf1ae9e9f63ed 2023-04-23 05:55:12,807 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61a97f406f2b42cd83baf1ae9e9f63ed, entries=150, sequenceid=934, filesize=11.9 K 2023-04-23 05:55:12,808 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/7a5989d95fe24349b394c3cae2a2f279 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7a5989d95fe24349b394c3cae2a2f279 2023-04-23 05:55:12,811 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7a5989d95fe24349b394c3cae2a2f279, entries=150, sequenceid=934, filesize=11.9 K 2023-04-23 05:55:12,812 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=93.93 KB/96180 for 911c079a878d9ede8370cba6d74e3efd in 45ms, sequenceid=934, compaction requested=true 2023-04-23 05:55:12,812 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:12,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:12,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:12,813 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:55:12,813 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:55:12,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:12,813 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:12,813 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-23 05:55:12,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:12,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:12,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:12,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:12,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:12,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:12,814 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:55:12,814 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/A is initiating minor compaction (all files) 2023-04-23 05:55:12,815 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/A in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:12,815 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3ee85ea75d5144aaa61cebeb2585efd3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/9d230443fc844d0eab78ab52f701d6f1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ba2cbccf7ef94b0bb7094da573c45ba5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c9c54bbca2b64cb9bb9488a0e4cdee5a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/edb2ec046eed4572bf0f54910f248f33] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=60.3 K 2023-04-23 05:55:12,815 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:55:12,815 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/B is initiating minor compaction (all files) 2023-04-23 05:55:12,815 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/B in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:12,815 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/b4f83cdb84424e929b737d9e47cfb374, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/5e461e1d272b4dcea32e0ef0c8d9bb6f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/55b9d7529b7346269d0a912ba4859573, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/cfebe44c4ede4a9abefdda3ec2b255ca, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61a97f406f2b42cd83baf1ae9e9f63ed] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=60.3 K 2023-04-23 05:55:12,815 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 3ee85ea75d5144aaa61cebeb2585efd3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1682229308260 2023-04-23 05:55:12,816 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b4f83cdb84424e929b737d9e47cfb374, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1682229308260 2023-04-23 05:55:12,816 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 5e461e1d272b4dcea32e0ef0c8d9bb6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=874, earliestPutTs=1682229308395 2023-04-23 05:55:12,816 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 9d230443fc844d0eab78ab52f701d6f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=874, earliestPutTs=1682229308395 2023-04-23 05:55:12,817 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 55b9d7529b7346269d0a912ba4859573, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=886, earliestPutTs=1682229309270 2023-04-23 05:55:12,817 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting ba2cbccf7ef94b0bb7094da573c45ba5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=886, earliestPutTs=1682229309270 2023-04-23 05:55:12,817 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting cfebe44c4ede4a9abefdda3ec2b255ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1682229310496 2023-04-23 05:55:12,817 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c9c54bbca2b64cb9bb9488a0e4cdee5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1682229310496 2023-04-23 05:55:12,817 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 61a97f406f2b42cd83baf1ae9e9f63ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1682229311636 2023-04-23 05:55:12,818 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting edb2ec046eed4572bf0f54910f248f33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1682229311636 2023-04-23 05:55:12,827 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#A#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:12,828 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#B#compaction#275 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:12,837 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/680025c3767e4538b0b46e08916f7d31 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/680025c3767e4538b0b46e08916f7d31 2023-04-23 05:55:12,842 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/B of 911c079a878d9ede8370cba6d74e3efd into 680025c3767e4538b0b46e08916f7d31(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:12,842 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:12,842 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/B, priority=11, startTime=1682229312813; duration=0sec 2023-04-23 05:55:12,843 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:12,843 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:55:12,844 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73832 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:55:12,844 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/C is initiating minor compaction (all files) 2023-04-23 05:55:12,844 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/C in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:12,845 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/79b8d45be0924d0e900044667de56b33, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/f7c881349e054ae78d369f8916691b68, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/d1d3ec1ca6ca479ebe48a4eeaa09b23a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9306f390719244b2afe2732e8208b305, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a4930698f49d4c98a8a8f419e7cd73e4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7a5989d95fe24349b394c3cae2a2f279] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=72.1 K 2023-04-23 05:55:12,845 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 79b8d45be0924d0e900044667de56b33, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=823, earliestPutTs=1682229307125 2023-04-23 05:55:12,845 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f7c881349e054ae78d369f8916691b68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=834, earliestPutTs=1682229308260 2023-04-23 05:55:12,846 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d1d3ec1ca6ca479ebe48a4eeaa09b23a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=874, earliestPutTs=1682229308395 2023-04-23 05:55:12,846 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9306f390719244b2afe2732e8208b305, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=886, earliestPutTs=1682229309270 2023-04-23 05:55:12,847 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a4930698f49d4c98a8a8f419e7cd73e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=923, earliestPutTs=1682229310496 2023-04-23 05:55:12,847 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7a5989d95fe24349b394c3cae2a2f279, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1682229311636 2023-04-23 05:55:12,856 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#C#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:12,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:12,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229372860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,867 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:12,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:12,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:12,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:12,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:12,870 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:12,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:12,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:12,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:12,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:12,871 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:12,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:12,872 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,872 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:12,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:12,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:12,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:12,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:12,873 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:12,873 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:12,873 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:12,873 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:12,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:12,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:12,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:12,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79bdb7d4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:12,877 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:12,877 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:12,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:12,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:12,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:12,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:12,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:12,883 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:12,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:12,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:12,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:12,883 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:12,884 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,884 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,884 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:12,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,885 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,886 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,888 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,888 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:12,888 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:12,888 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:12,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:12,888 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:12,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:12,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:12,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:12,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:12,889 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:12,889 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:12,889 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:12,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:12,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229372963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:12,989 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:12,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:12,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:12,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:12,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:12,993 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:12,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:12,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:12,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:12,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:12,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:13,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:13,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:13,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:13,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:13,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:13,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:13,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:13,003 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,003 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:13,004 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:13,004 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:13,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:13,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:13,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:13,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:13,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b1ca7a8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:13,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:13,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:13,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:13,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:13,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:13,010 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:13,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:13,013 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:13,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:13,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:13,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:13,013 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:13,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,016 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:13,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:13,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:13,020 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,020 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:13,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,020 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:13,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,120 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:13,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:13,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:13,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:13,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:13,124 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:13,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:13,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:13,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,126 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:13,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:13,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:13,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:13,127 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:13,128 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,128 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:13,128 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:13,128 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:13,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:13,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:13,128 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:13,129 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,129 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,130 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,130 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:13,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54393c52[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:13,133 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:13,133 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,135 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,135 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:13,136 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:13,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:13,136 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:13,137 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:13,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:13,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:13,139 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:13,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:13,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:13,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:13,139 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:13,141 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,141 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,141 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,143 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,145 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,145 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,145 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,145 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,145 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:13,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:13,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:13,146 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,146 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,146 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,146 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,146 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:13,146 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:13,146 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:13,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229373166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,224 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=952 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/cbed8e2b7927480ebb7351479cbb90ec 2023-04-23 05:55:13,236 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=952 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/ec96705d5f474c538ca019bd660059c9 2023-04-23 05:55:13,237 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/c685f5d5e0274d2e9c1ea2a10218167a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c685f5d5e0274d2e9c1ea2a10218167a 2023-04-23 05:55:13,243 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/A of 911c079a878d9ede8370cba6d74e3efd into c685f5d5e0274d2e9c1ea2a10218167a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:13,243 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:13,243 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/A, priority=11, startTime=1682229312812; duration=0sec 2023-04-23 05:55:13,243 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:13,246 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:13,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:13,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:13,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:13,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:13,250 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:13,250 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,250 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:13,250 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:13,250 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,250 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,253 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,253 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,253 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,253 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:13,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:13,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:13,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:13,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:13,254 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:13,254 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,254 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:13,254 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:13,254 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:13,255 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,255 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:13,255 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:13,255 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:13,256 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,256 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,257 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,257 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:13,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2146cae5[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:13,258 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:13,258 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,259 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,259 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:13,259 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:13,259 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:13,259 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,259 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:13,261 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,261 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:13,261 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,261 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:13,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:13,264 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:13,264 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:13,264 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:13,264 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:13,264 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:13,265 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/0ed31b6e0c1b494a9bd602a18d60ee3a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0ed31b6e0c1b494a9bd602a18d60ee3a 2023-04-23 05:55:13,267 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,267 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,267 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,268 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,270 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,271 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/C of 911c079a878d9ede8370cba6d74e3efd into 0ed31b6e0c1b494a9bd602a18d60ee3a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:13,271 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:13,271 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/C, priority=10, startTime=1682229312813; duration=0sec 2023-04-23 05:55:13,271 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,273 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,274 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:13,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:13,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,274 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:13,274 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:13,374 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:13,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:13,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:13,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:13,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:13,378 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:13,378 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,378 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:13,378 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:13,378 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,378 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,380 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,380 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,380 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:13,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:13,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:13,382 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,382 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:13,382 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:13,382 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:13,382 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:13,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,387 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,387 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,387 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:13,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3551ca0e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:13,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:13,389 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,390 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:13,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:13,391 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,391 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,392 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,392 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:13,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:13,394 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:13,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:13,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:13,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:13,395 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:13,396 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,396 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,396 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:13,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:13,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:13,402 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,401 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,402 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:13,402 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:13,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,402 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,402 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,402 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,402 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:13,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229373468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,502 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:13,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:13,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:13,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:13,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:13,506 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:13,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:13,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:13,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,508 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:13,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:13,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:13,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:13,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:13,509 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:13,509 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,509 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:13,510 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,510 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:13,510 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:13,510 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:13,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:13,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,512 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,512 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:13,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bae1aeb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:13,513 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:13,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,515 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:13,515 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:13,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:13,515 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:13,517 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,517 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:13,517 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,517 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:13,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:13,520 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:13,520 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:13,520 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:13,520 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:13,520 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:13,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,522 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,526 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:13,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:13,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:13,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,526 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:13,526 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:13,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,527 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,627 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:13,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:13,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:13,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:13,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:13,630 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:13,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:13,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:13,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,632 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,632 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,632 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:13,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:13,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:13,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:13,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:13,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:13,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:13,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:13,634 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,634 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:13,634 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:13,634 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:13,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:13,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:13,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:13,637 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,637 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,637 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,637 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:13,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:13,639 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:13,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e86060[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:13,639 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:13,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:13,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:13,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:13,642 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:13,642 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,642 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:13,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:13,645 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:13,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:13,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:13,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:13,645 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:13,646 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,646 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,646 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:13,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=952 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/a102360294644ee9ae054feb9c4bb23d 2023-04-23 05:55:13,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:13,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:13,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:13,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/cbed8e2b7927480ebb7351479cbb90ec as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cbed8e2b7927480ebb7351479cbb90ec 2023-04-23 05:55:13,654 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,654 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:13,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cbed8e2b7927480ebb7351479cbb90ec, entries=200, sequenceid=952, filesize=14.3 K 2023-04-23 05:55:13,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/ec96705d5f474c538ca019bd660059c9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec96705d5f474c538ca019bd660059c9 2023-04-23 05:55:13,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:13,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:13,658 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,658 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,658 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,658 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,658 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:13,658 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,658 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:13,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:13,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:13,658 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,659 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:13,659 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:13,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec96705d5f474c538ca019bd660059c9, entries=150, sequenceid=952, filesize=11.9 K 2023-04-23 05:55:13,663 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/a102360294644ee9ae054feb9c4bb23d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a102360294644ee9ae054feb9c4bb23d 2023-04-23 05:55:13,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a102360294644ee9ae054feb9c4bb23d, entries=150, sequenceid=952, filesize=11.9 K 2023-04-23 05:55:13,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 911c079a878d9ede8370cba6d74e3efd in 855ms, sequenceid=952, compaction requested=false 2023-04-23 05:55:13,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:13,759 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:13,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:13,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:13,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:13,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:13,762 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:13,762 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,762 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:13,762 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:13,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:13,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:13,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:13,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:13,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:13,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:13,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:13,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:13,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:13,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:13,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:13,765 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:13,766 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:13,766 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:13,766 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:13,766 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:13,766 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:13,766 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-23 05:55:13,766 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:13,766 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:13,766 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:13,766 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:13,766 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:13,767 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:13,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:13,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:14,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:14,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229374006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:14,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229374109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,178 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=969 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/74f6be1b6d1d461aa64953fe49e6369e 2023-04-23 05:55:14,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:14,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229374311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,588 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=969 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/f98b6ef9d7774a23a60be490c4f9cffa 2023-04-23 05:55:14,598 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=969 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/0b61c7cb08c949f1937338911140434a 2023-04-23 05:55:14,602 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/74f6be1b6d1d461aa64953fe49e6369e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/74f6be1b6d1d461aa64953fe49e6369e 2023-04-23 05:55:14,606 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/74f6be1b6d1d461aa64953fe49e6369e, entries=150, sequenceid=969, filesize=11.9 K 2023-04-23 05:55:14,606 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/f98b6ef9d7774a23a60be490c4f9cffa as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f98b6ef9d7774a23a60be490c4f9cffa 2023-04-23 05:55:14,611 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f98b6ef9d7774a23a60be490c4f9cffa, entries=150, sequenceid=969, filesize=11.9 K 2023-04-23 05:55:14,611 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/0b61c7cb08c949f1937338911140434a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0b61c7cb08c949f1937338911140434a 2023-04-23 05:55:14,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:14,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229374613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,617 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0b61c7cb08c949f1937338911140434a, entries=150, sequenceid=969, filesize=11.9 K 2023-04-23 05:55:14,618 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 911c079a878d9ede8370cba6d74e3efd in 852ms, sequenceid=969, compaction requested=true 2023-04-23 05:55:14,618 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:14,618 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:14,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:14,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:14,618 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:14,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:14,623 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:14,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:14,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:14,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:14,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:14,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:14,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@616e8dba[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:14,626 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:14,626 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:14,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:14,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:14,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:14,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:14,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:14,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:14,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:14,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:14,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:14,633 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:14,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:14,636 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:14,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:14,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:14,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:14,636 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:14,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:14,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:14,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:14,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:14,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:14,639 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:14,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:14,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:14,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,643 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:14,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:14,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:14,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:14,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:14,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:14,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:14,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:14,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:14,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:14,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:14,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:14,644 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:14,644 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:14,744 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:14,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:14,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:14,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:14,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:14,748 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:14,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:14,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:14,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:14,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:14,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:14,750 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:14,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:14,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:14,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:14,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:14,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:14,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:14,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:14,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:14,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:14,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:14,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:14,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:14,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:14,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:14,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. started... 2023-04-23 05:55:14,752 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-23 05:55:14,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:14,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:14,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:14,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:14,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:14,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:15,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:15,121 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. as already flushing 2023-04-23 05:55:15,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:15,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229375154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,161 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=987 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/0e0e5e9c7a474638a6bde76f62ddcf44 2023-04-23 05:55:15,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:15,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229375257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:15,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:52826 deadline: 1682229375462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,571 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=987 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/9304ef4d77f04d76be113b2bbfde312e 2023-04-23 05:55:15,580 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=987 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/163586716e624e09ae0ed07ebdd26357 2023-04-23 05:55:15,584 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/0e0e5e9c7a474638a6bde76f62ddcf44 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/0e0e5e9c7a474638a6bde76f62ddcf44 2023-04-23 05:55:15,587 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/0e0e5e9c7a474638a6bde76f62ddcf44, entries=150, sequenceid=987, filesize=11.9 K 2023-04-23 05:55:15,588 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/9304ef4d77f04d76be113b2bbfde312e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9304ef4d77f04d76be113b2bbfde312e 2023-04-23 05:55:15,591 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9304ef4d77f04d76be113b2bbfde312e, entries=150, sequenceid=987, filesize=11.9 K 2023-04-23 05:55:15,592 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/163586716e624e09ae0ed07ebdd26357 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/163586716e624e09ae0ed07ebdd26357 2023-04-23 05:55:15,595 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/163586716e624e09ae0ed07ebdd26357, entries=150, sequenceid=987, filesize=11.9 K 2023-04-23 05:55:15,596 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 911c079a878d9ede8370cba6d74e3efd in 844ms, sequenceid=987, compaction requested=true 2023-04-23 05:55:15,596 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:15,596 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:15,596 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:15,596 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:15,597 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:15,600 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:15,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:15,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:15,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:15,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a543a23[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:15,602 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:15,602 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:15,603 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:15,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:15,603 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:15,603 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:15,603 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:15,603 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,603 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:15,604 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:15,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,604 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:15,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:15,604 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:15,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:15,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:15,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:15,607 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:15,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:15,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:15,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:15,607 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:15,609 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:15,609 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:15,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:15,609 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:15,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:15,610 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,611 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:15,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:15,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:15,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:15,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:15,614 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:15,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:15,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:15,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:15,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:15,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:15,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:15,615 DEBUG [Thread-1551] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:15,615 DEBUG [Thread-1551] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:15,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:15,627 DEBUG [Thread-1558] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b4fa5d3 to 127.0.0.1:63802 2023-04-23 05:55:15,627 DEBUG [Thread-1558] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:15,627 DEBUG [Thread-1556] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x681ace6c to 127.0.0.1:63802 2023-04-23 05:55:15,627 DEBUG [Thread-1556] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:15,628 DEBUG [Thread-1554] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f29c1c4 to 127.0.0.1:63802 2023-04-23 05:55:15,628 DEBUG [Thread-1554] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:15,629 DEBUG [Thread-1552] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e5b1c14 to 127.0.0.1:63802 2023-04-23 05:55:15,629 DEBUG [Thread-1552] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:15,629 DEBUG [Thread-1560] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45de5ad8 to 127.0.0.1:63802 2023-04-23 05:55:15,629 DEBUG [Thread-1560] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:15,715 DEBUG [Thread-1551] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:15,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:15,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:15,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-23 05:55:15,768 DEBUG [Thread-1549] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6705d115 to 127.0.0.1:63802 2023-04-23 05:55:15,768 DEBUG [Thread-1549] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:15,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:15,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:15,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:15,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:15,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:15,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:16,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1002 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/ecbe44ed518c42728fb0c3fa074d234a 2023-04-23 05:55:16,551 DEBUG [Thread-1547] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3562c7bf to 127.0.0.1:63802 2023-04-23 05:55:16,551 DEBUG [Thread-1547] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:16,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1002 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/578a295e00a84b69b28b396d359e8f28 2023-04-23 05:55:16,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1002 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/72cc03dac4c04ba5ac186d88ec0c30ea 2023-04-23 05:55:16,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/ecbe44ed518c42728fb0c3fa074d234a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ecbe44ed518c42728fb0c3fa074d234a 2023-04-23 05:55:17,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ecbe44ed518c42728fb0c3fa074d234a, entries=150, sequenceid=1002, filesize=11.9 K 2023-04-23 05:55:17,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/578a295e00a84b69b28b396d359e8f28 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/578a295e00a84b69b28b396d359e8f28 2023-04-23 05:55:17,005 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/578a295e00a84b69b28b396d359e8f28, entries=150, sequenceid=1002, filesize=11.9 K 2023-04-23 05:55:17,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/72cc03dac4c04ba5ac186d88ec0c30ea as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/72cc03dac4c04ba5ac186d88ec0c30ea 2023-04-23 05:55:17,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/72cc03dac4c04ba5ac186d88ec0c30ea, entries=150, sequenceid=1002, filesize=11.9 K 2023-04-23 05:55:17,009 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=6.71 KB/6870 for 911c079a878d9ede8370cba6d74e3efd in 1241ms, sequenceid=1002, compaction requested=true 2023-04-23 05:55:17,010 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:17,010 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:17,010 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:17,010 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:55:17,010 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:55:17,010 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:17,011 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64324 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:55:17,011 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:55:17,011 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/A is initiating minor compaction (all files) 2023-04-23 05:55:17,011 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/B is initiating minor compaction (all files) 2023-04-23 05:55:17,011 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/A in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:17,011 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/B in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:17,012 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c685f5d5e0274d2e9c1ea2a10218167a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cbed8e2b7927480ebb7351479cbb90ec, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/74f6be1b6d1d461aa64953fe49e6369e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/0e0e5e9c7a474638a6bde76f62ddcf44, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ecbe44ed518c42728fb0c3fa074d234a] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=62.8 K 2023-04-23 05:55:17,012 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/680025c3767e4538b0b46e08916f7d31, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec96705d5f474c538ca019bd660059c9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f98b6ef9d7774a23a60be490c4f9cffa, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9304ef4d77f04d76be113b2bbfde312e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/578a295e00a84b69b28b396d359e8f28] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=60.4 K 2023-04-23 05:55:17,012 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c685f5d5e0274d2e9c1ea2a10218167a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1682229311636 2023-04-23 05:55:17,012 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 680025c3767e4538b0b46e08916f7d31, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1682229311636 2023-04-23 05:55:17,012 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting cbed8e2b7927480ebb7351479cbb90ec, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=952, earliestPutTs=1682229312797 2023-04-23 05:55:17,012 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ec96705d5f474c538ca019bd660059c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=952, earliestPutTs=1682229312797 2023-04-23 05:55:17,013 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 74f6be1b6d1d461aa64953fe49e6369e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1682229312815 2023-04-23 05:55:17,013 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f98b6ef9d7774a23a60be490c4f9cffa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1682229312815 2023-04-23 05:55:17,013 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 0e0e5e9c7a474638a6bde76f62ddcf44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1682229313994 2023-04-23 05:55:17,013 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9304ef4d77f04d76be113b2bbfde312e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1682229313994 2023-04-23 05:55:17,013 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting ecbe44ed518c42728fb0c3fa074d234a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1002, earliestPutTs=1682229315147 2023-04-23 05:55:17,013 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 578a295e00a84b69b28b396d359e8f28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1002, earliestPutTs=1682229315147 2023-04-23 05:55:17,021 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#A#compaction#289 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:17,021 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#B#compaction#290 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:17,429 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/b1fe1256be5749df9403e755a5405316 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/b1fe1256be5749df9403e755a5405316 2023-04-23 05:55:17,430 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/7873b662248743e0a23b80a6dc0ea814 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/7873b662248743e0a23b80a6dc0ea814 2023-04-23 05:55:17,434 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/A of 911c079a878d9ede8370cba6d74e3efd into 7873b662248743e0a23b80a6dc0ea814(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:17,434 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/B of 911c079a878d9ede8370cba6d74e3efd into b1fe1256be5749df9403e755a5405316(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:17,434 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:17,434 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:17,434 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/A, priority=11, startTime=1682229317010; duration=0sec 2023-04-23 05:55:17,434 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/B, priority=11, startTime=1682229317010; duration=0sec 2023-04-23 05:55:17,434 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:17,434 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:17,434 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:55:17,435 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:55:17,435 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 911c079a878d9ede8370cba6d74e3efd/C is initiating minor compaction (all files) 2023-04-23 05:55:17,435 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 911c079a878d9ede8370cba6d74e3efd/C in TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:17,435 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0ed31b6e0c1b494a9bd602a18d60ee3a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a102360294644ee9ae054feb9c4bb23d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0b61c7cb08c949f1937338911140434a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/163586716e624e09ae0ed07ebdd26357, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/72cc03dac4c04ba5ac186d88ec0c30ea] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp, totalSize=60.4 K 2023-04-23 05:55:17,436 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 0ed31b6e0c1b494a9bd602a18d60ee3a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=934, earliestPutTs=1682229311636 2023-04-23 05:55:17,436 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a102360294644ee9ae054feb9c4bb23d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=952, earliestPutTs=1682229312797 2023-04-23 05:55:17,436 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 0b61c7cb08c949f1937338911140434a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=969, earliestPutTs=1682229312815 2023-04-23 05:55:17,436 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 163586716e624e09ae0ed07ebdd26357, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=987, earliestPutTs=1682229313994 2023-04-23 05:55:17,437 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 72cc03dac4c04ba5ac186d88ec0c30ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1002, earliestPutTs=1682229315147 2023-04-23 05:55:17,443 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 911c079a878d9ede8370cba6d74e3efd#C#compaction#291 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:17,851 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/99f06a321bb44bcd806ca84f34b266bd as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/99f06a321bb44bcd806ca84f34b266bd 2023-04-23 05:55:17,855 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 911c079a878d9ede8370cba6d74e3efd/C of 911c079a878d9ede8370cba6d74e3efd into 99f06a321bb44bcd806ca84f34b266bd(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:17,855 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:17,855 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd., storeName=911c079a878d9ede8370cba6d74e3efd/C, priority=11, startTime=1682229317010; duration=0sec 2023-04-23 05:55:17,855 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:19,575 DEBUG [Thread-1545] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5a81fa to 127.0.0.1:63802 2023-04-23 05:55:19,575 DEBUG [Thread-1545] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:19,629 DEBUG [Thread-1541] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13544701 to 127.0.0.1:63802 2023-04-23 05:55:19,629 DEBUG [Thread-1541] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:19,683 DEBUG [Thread-1543] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10b79c2c to 127.0.0.1:63802 2023-04-23 05:55:19,683 DEBUG [Thread-1543] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:19,683 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-23 05:55:19,683 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-23 05:55:19,683 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 129 2023-04-23 05:55:19,683 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 122 2023-04-23 05:55:19,683 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 258 2023-04-23 05:55:19,684 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 311 2023-04-23 05:55:19,684 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-23 05:55:19,684 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 5656 2023-04-23 05:55:19,684 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 5598 2023-04-23 05:55:19,684 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 5578 2023-04-23 05:55:19,684 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 5688 2023-04-23 05:55:19,684 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 5599 2023-04-23 05:55:19,684 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-23 05:55:19,684 INFO [Listener at localhost/41845] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-23 05:55:19,684 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53921272 to 127.0.0.1:63802 2023-04-23 05:55:19,684 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:19,685 INFO [Listener at localhost/41845] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-23 05:55:19,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-23 05:55:19,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:19,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-23 05:55:19,688 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229319688"}]},"ts":"1682229319688"} 2023-04-23 05:55:19,689 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-23 05:55:19,690 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-23 05:55:19,691 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=911c079a878d9ede8370cba6d74e3efd, UNASSIGN}] 2023-04-23 05:55:19,692 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=911c079a878d9ede8370cba6d74e3efd, UNASSIGN 2023-04-23 05:55:19,692 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=911c079a878d9ede8370cba6d74e3efd, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:19,692 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229319692"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229319692"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229319692"}]},"ts":"1682229319692"} 2023-04-23 05:55:19,693 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:55:19,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-23 05:55:19,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:19,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 911c079a878d9ede8370cba6d74e3efd, disabling compactions & flushes 2023-04-23 05:55:19,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:19,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:19,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. after waiting 0 ms 2023-04-23 05:55:19,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:19,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 911c079a878d9ede8370cba6d74e3efd 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-23 05:55:19,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=A 2023-04-23 05:55:19,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:19,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=B 2023-04-23 05:55:19,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:19,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 911c079a878d9ede8370cba6d74e3efd, store=C 2023-04-23 05:55:19,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:19,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-23 05:55:20,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1012 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/c00bc4d6764c445d8d4abf5699294949 2023-04-23 05:55:20,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-23 05:55:20,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1012 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/86f831720a304050bb65e78d232dc8c6 2023-04-23 05:55:20,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-23 05:55:21,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1012 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/beb7fc57e4e945bd9136b70e20150a18 2023-04-23 05:55:21,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/A/c00bc4d6764c445d8d4abf5699294949 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c00bc4d6764c445d8d4abf5699294949 2023-04-23 05:55:21,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c00bc4d6764c445d8d4abf5699294949, entries=100, sequenceid=1012, filesize=9.5 K 2023-04-23 05:55:21,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/B/86f831720a304050bb65e78d232dc8c6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/86f831720a304050bb65e78d232dc8c6 2023-04-23 05:55:21,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/86f831720a304050bb65e78d232dc8c6, entries=100, sequenceid=1012, filesize=9.5 K 2023-04-23 05:55:21,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/.tmp/C/beb7fc57e4e945bd9136b70e20150a18 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/beb7fc57e4e945bd9136b70e20150a18 2023-04-23 05:55:21,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/beb7fc57e4e945bd9136b70e20150a18, entries=100, sequenceid=1012, filesize=9.5 K 2023-04-23 05:55:21,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 911c079a878d9ede8370cba6d74e3efd in 1243ms, sequenceid=1012, compaction requested=false 2023-04-23 05:55:21,109 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/952f5966f5e8401582e1233a7fddf233, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/416217d6429141af9cc12dd224e31e2b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/e0d6d4797cb8452bba1215755d0b1766, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/8da22d8a626d409c831d20d4d8fd01f9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3059e55e5f48480693697a18c1ac9a43, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/fd218d14a9414d62ad579c16f6d44ee2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/75ea2035da8d4c6b9d2f8ad8ee5abd45, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/52bf6627e70c4b5786dd05dae8223f5a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/83bbe7dde6204f219db5471592705063, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b95d0f9976ed4b4ebfc324290f331e56, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/924846bd3bc94cfbb0f6154a690f8798, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/54e71d6b634245118abff43f90afb512, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f9ea77137d8c47d49221749b0b1856fc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/a704432cb4b5443b8a7d91fcff0d394b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2811b93928849d2bdaeb18cfc7f0435, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/dde6f51a6a874435beb7896f2fb9527e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cfc511f9b7d349979686e3c010749e86, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f81f9c80db7b4bd7a37d467532d595ef, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/581c92003eb0441fa375744cb713edb5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f6cc92c297be4f7386f62597705e66be, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2a654cc565944f98a3f6d64a7d086c8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c73e36b22ff64e01b1c59e08bf043023, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/605913b5f2ad4ca78564aa39de89a1f1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3ee85ea75d5144aaa61cebeb2585efd3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/9d230443fc844d0eab78ab52f701d6f1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ba2cbccf7ef94b0bb7094da573c45ba5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c9c54bbca2b64cb9bb9488a0e4cdee5a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c685f5d5e0274d2e9c1ea2a10218167a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/edb2ec046eed4572bf0f54910f248f33, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cbed8e2b7927480ebb7351479cbb90ec, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/74f6be1b6d1d461aa64953fe49e6369e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/0e0e5e9c7a474638a6bde76f62ddcf44, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ecbe44ed518c42728fb0c3fa074d234a] to archive 2023-04-23 05:55:21,110 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:55:21,112 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/952f5966f5e8401582e1233a7fddf233 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/952f5966f5e8401582e1233a7fddf233 2023-04-23 05:55:21,113 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/416217d6429141af9cc12dd224e31e2b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/416217d6429141af9cc12dd224e31e2b 2023-04-23 05:55:21,114 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/e0d6d4797cb8452bba1215755d0b1766 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/e0d6d4797cb8452bba1215755d0b1766 2023-04-23 05:55:21,114 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/8da22d8a626d409c831d20d4d8fd01f9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/8da22d8a626d409c831d20d4d8fd01f9 2023-04-23 05:55:21,115 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3059e55e5f48480693697a18c1ac9a43 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3059e55e5f48480693697a18c1ac9a43 2023-04-23 05:55:21,116 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/fd218d14a9414d62ad579c16f6d44ee2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/fd218d14a9414d62ad579c16f6d44ee2 2023-04-23 05:55:21,117 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/75ea2035da8d4c6b9d2f8ad8ee5abd45 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/75ea2035da8d4c6b9d2f8ad8ee5abd45 2023-04-23 05:55:21,117 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/52bf6627e70c4b5786dd05dae8223f5a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/52bf6627e70c4b5786dd05dae8223f5a 2023-04-23 05:55:21,118 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/83bbe7dde6204f219db5471592705063 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/83bbe7dde6204f219db5471592705063 2023-04-23 05:55:21,119 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b95d0f9976ed4b4ebfc324290f331e56 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b95d0f9976ed4b4ebfc324290f331e56 2023-04-23 05:55:21,120 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/924846bd3bc94cfbb0f6154a690f8798 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/924846bd3bc94cfbb0f6154a690f8798 2023-04-23 05:55:21,120 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/54e71d6b634245118abff43f90afb512 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/54e71d6b634245118abff43f90afb512 2023-04-23 05:55:21,121 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f9ea77137d8c47d49221749b0b1856fc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f9ea77137d8c47d49221749b0b1856fc 2023-04-23 05:55:21,122 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/a704432cb4b5443b8a7d91fcff0d394b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/a704432cb4b5443b8a7d91fcff0d394b 2023-04-23 05:55:21,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2811b93928849d2bdaeb18cfc7f0435 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2811b93928849d2bdaeb18cfc7f0435 2023-04-23 05:55:21,123 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/dde6f51a6a874435beb7896f2fb9527e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/dde6f51a6a874435beb7896f2fb9527e 2023-04-23 05:55:21,124 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cfc511f9b7d349979686e3c010749e86 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cfc511f9b7d349979686e3c010749e86 2023-04-23 05:55:21,125 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f81f9c80db7b4bd7a37d467532d595ef to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f81f9c80db7b4bd7a37d467532d595ef 2023-04-23 05:55:21,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/581c92003eb0441fa375744cb713edb5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/581c92003eb0441fa375744cb713edb5 2023-04-23 05:55:21,126 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f6cc92c297be4f7386f62597705e66be to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/f6cc92c297be4f7386f62597705e66be 2023-04-23 05:55:21,127 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2a654cc565944f98a3f6d64a7d086c8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/b2a654cc565944f98a3f6d64a7d086c8 2023-04-23 05:55:21,128 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c73e36b22ff64e01b1c59e08bf043023 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c73e36b22ff64e01b1c59e08bf043023 2023-04-23 05:55:21,128 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/605913b5f2ad4ca78564aa39de89a1f1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/605913b5f2ad4ca78564aa39de89a1f1 2023-04-23 05:55:21,129 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3ee85ea75d5144aaa61cebeb2585efd3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/3ee85ea75d5144aaa61cebeb2585efd3 2023-04-23 05:55:21,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/9d230443fc844d0eab78ab52f701d6f1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/9d230443fc844d0eab78ab52f701d6f1 2023-04-23 05:55:21,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ba2cbccf7ef94b0bb7094da573c45ba5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ba2cbccf7ef94b0bb7094da573c45ba5 2023-04-23 05:55:21,131 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c9c54bbca2b64cb9bb9488a0e4cdee5a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c9c54bbca2b64cb9bb9488a0e4cdee5a 2023-04-23 05:55:21,132 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c685f5d5e0274d2e9c1ea2a10218167a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c685f5d5e0274d2e9c1ea2a10218167a 2023-04-23 05:55:21,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/edb2ec046eed4572bf0f54910f248f33 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/edb2ec046eed4572bf0f54910f248f33 2023-04-23 05:55:21,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cbed8e2b7927480ebb7351479cbb90ec to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/cbed8e2b7927480ebb7351479cbb90ec 2023-04-23 05:55:21,134 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/74f6be1b6d1d461aa64953fe49e6369e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/74f6be1b6d1d461aa64953fe49e6369e 2023-04-23 05:55:21,135 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/0e0e5e9c7a474638a6bde76f62ddcf44 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/0e0e5e9c7a474638a6bde76f62ddcf44 2023-04-23 05:55:21,136 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ecbe44ed518c42728fb0c3fa074d234a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/ecbe44ed518c42728fb0c3fa074d234a 2023-04-23 05:55:21,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fa923bced2a94633bc343c1a71557cd5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec2c02fc590c4408a17654639f93a413, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/e515e9c08b2a432f925462896f08f187, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/0df1c0a6b9ad4c16ada7b6953356aeab, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6a89a7ca8734473c90c6638a29bbcc91, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f19082223ea94e0a9394c337f74f7778, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6f15710bc080404dab1196b6765430bc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cf3ea15aa944db19b1cab8f364d7900, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cedf91eb0ff4d7485ea0e6aeac74aa8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7249124d98d44bc5b81a6699024284d6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2481978c26274a18bba24392cb9ccf72, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/c1090617b5c8420aa8897c399c78b44b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/457184c99f174749a13a187c8c62260b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/8683167810e04f849fd7cd492a500793, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2eda4e2be29c4dec9855022c582457cb, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/3e5e9cb7e9a84cfaa003c9cfeb805d87, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/82b41b740d0b46088ae872dc3cf40299, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9a668450096f482bb33a114afb4e641a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7c0ad94e3ad34ad69d96add734725fa3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/75f0f2753e0043e29a2a488681b79d31, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fadc03b7b64b49e7941aaa8286a9b3ba, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61496f1c8205478fb3e74102cd9196e6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/b4f83cdb84424e929b737d9e47cfb374, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/bea8237781f34c7ba6620a29e176820e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/5e461e1d272b4dcea32e0ef0c8d9bb6f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/55b9d7529b7346269d0a912ba4859573, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/cfebe44c4ede4a9abefdda3ec2b255ca, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/680025c3767e4538b0b46e08916f7d31, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61a97f406f2b42cd83baf1ae9e9f63ed, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec96705d5f474c538ca019bd660059c9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f98b6ef9d7774a23a60be490c4f9cffa, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9304ef4d77f04d76be113b2bbfde312e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/578a295e00a84b69b28b396d359e8f28] to archive 2023-04-23 05:55:21,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:55:21,141 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fa923bced2a94633bc343c1a71557cd5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fa923bced2a94633bc343c1a71557cd5 2023-04-23 05:55:21,142 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec2c02fc590c4408a17654639f93a413 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec2c02fc590c4408a17654639f93a413 2023-04-23 05:55:21,143 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/e515e9c08b2a432f925462896f08f187 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/e515e9c08b2a432f925462896f08f187 2023-04-23 05:55:21,144 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/0df1c0a6b9ad4c16ada7b6953356aeab to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/0df1c0a6b9ad4c16ada7b6953356aeab 2023-04-23 05:55:21,145 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6a89a7ca8734473c90c6638a29bbcc91 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6a89a7ca8734473c90c6638a29bbcc91 2023-04-23 05:55:21,146 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f19082223ea94e0a9394c337f74f7778 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f19082223ea94e0a9394c337f74f7778 2023-04-23 05:55:21,147 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6f15710bc080404dab1196b6765430bc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/6f15710bc080404dab1196b6765430bc 2023-04-23 05:55:21,147 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cf3ea15aa944db19b1cab8f364d7900 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cf3ea15aa944db19b1cab8f364d7900 2023-04-23 05:55:21,148 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cedf91eb0ff4d7485ea0e6aeac74aa8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2cedf91eb0ff4d7485ea0e6aeac74aa8 2023-04-23 05:55:21,149 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7249124d98d44bc5b81a6699024284d6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7249124d98d44bc5b81a6699024284d6 2023-04-23 05:55:21,150 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2481978c26274a18bba24392cb9ccf72 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2481978c26274a18bba24392cb9ccf72 2023-04-23 05:55:21,151 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/c1090617b5c8420aa8897c399c78b44b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/c1090617b5c8420aa8897c399c78b44b 2023-04-23 05:55:21,152 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/457184c99f174749a13a187c8c62260b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/457184c99f174749a13a187c8c62260b 2023-04-23 05:55:21,154 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/8683167810e04f849fd7cd492a500793 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/8683167810e04f849fd7cd492a500793 2023-04-23 05:55:21,155 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2eda4e2be29c4dec9855022c582457cb to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/2eda4e2be29c4dec9855022c582457cb 2023-04-23 05:55:21,156 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/3e5e9cb7e9a84cfaa003c9cfeb805d87 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/3e5e9cb7e9a84cfaa003c9cfeb805d87 2023-04-23 05:55:21,157 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/82b41b740d0b46088ae872dc3cf40299 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/82b41b740d0b46088ae872dc3cf40299 2023-04-23 05:55:21,158 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9a668450096f482bb33a114afb4e641a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9a668450096f482bb33a114afb4e641a 2023-04-23 05:55:21,158 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7c0ad94e3ad34ad69d96add734725fa3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/7c0ad94e3ad34ad69d96add734725fa3 2023-04-23 05:55:21,159 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/75f0f2753e0043e29a2a488681b79d31 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/75f0f2753e0043e29a2a488681b79d31 2023-04-23 05:55:21,160 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fadc03b7b64b49e7941aaa8286a9b3ba to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/fadc03b7b64b49e7941aaa8286a9b3ba 2023-04-23 05:55:21,161 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61496f1c8205478fb3e74102cd9196e6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61496f1c8205478fb3e74102cd9196e6 2023-04-23 05:55:21,162 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/b4f83cdb84424e929b737d9e47cfb374 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/b4f83cdb84424e929b737d9e47cfb374 2023-04-23 05:55:21,163 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/bea8237781f34c7ba6620a29e176820e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/bea8237781f34c7ba6620a29e176820e 2023-04-23 05:55:21,164 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/5e461e1d272b4dcea32e0ef0c8d9bb6f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/5e461e1d272b4dcea32e0ef0c8d9bb6f 2023-04-23 05:55:21,164 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/55b9d7529b7346269d0a912ba4859573 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/55b9d7529b7346269d0a912ba4859573 2023-04-23 05:55:21,165 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/cfebe44c4ede4a9abefdda3ec2b255ca to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/cfebe44c4ede4a9abefdda3ec2b255ca 2023-04-23 05:55:21,166 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/680025c3767e4538b0b46e08916f7d31 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/680025c3767e4538b0b46e08916f7d31 2023-04-23 05:55:21,167 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61a97f406f2b42cd83baf1ae9e9f63ed to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/61a97f406f2b42cd83baf1ae9e9f63ed 2023-04-23 05:55:21,167 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec96705d5f474c538ca019bd660059c9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/ec96705d5f474c538ca019bd660059c9 2023-04-23 05:55:21,168 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f98b6ef9d7774a23a60be490c4f9cffa to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/f98b6ef9d7774a23a60be490c4f9cffa 2023-04-23 05:55:21,169 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9304ef4d77f04d76be113b2bbfde312e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/9304ef4d77f04d76be113b2bbfde312e 2023-04-23 05:55:21,170 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/578a295e00a84b69b28b396d359e8f28 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/578a295e00a84b69b28b396d359e8f28 2023-04-23 05:55:21,173 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/87498d8d9df44d829839b75ab9cb514a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/acb3991d58004197b238a56ec676c568, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/031f33cc2ec0496cbfcf7ef43421ff4c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ab99a66694a14def97117c3d1f1c3e27, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/2f85da106031487faa506667dd269db0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/23cd8a6d424f4adfbeb0a1cc0f0c5117, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca60849574624bcf822bded65673c3a9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7b7fd4c83ad14515bdb442750a6bb6cf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a0aa8f11cf484f37863b0f0334715fb4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/51de38dfd08644eaaecadc56b0e0b49c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/1bcc6505af704cd0bfe4a3b2f88cc330, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/17ecbcceb2fc4705a7a5585bbdf0f6bc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9ebfacec72034f00af6e733b18404a50, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/410dfa6905934fafb240c5adfaf70203, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/78c75f9332bf42be9778e1feecd641ca, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a5e72b1c54b44567acab6eb1d27b7140, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0a17a6f7d58f47309acb2e6a97133597, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/eae45b335b83451ebe03783a81197d35, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/48a7d4352d8b43d8ad3ae684325882ba, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/725d83ae159647de92bb45ac770212d7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/79b8d45be0924d0e900044667de56b33, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca41f593090c4a2bbe0c4ced4c202b49, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/f7c881349e054ae78d369f8916691b68, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/d1d3ec1ca6ca479ebe48a4eeaa09b23a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9306f390719244b2afe2732e8208b305, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a4930698f49d4c98a8a8f419e7cd73e4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0ed31b6e0c1b494a9bd602a18d60ee3a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7a5989d95fe24349b394c3cae2a2f279, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a102360294644ee9ae054feb9c4bb23d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0b61c7cb08c949f1937338911140434a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/163586716e624e09ae0ed07ebdd26357, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/72cc03dac4c04ba5ac186d88ec0c30ea] to archive 2023-04-23 05:55:21,173 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:55:21,174 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/87498d8d9df44d829839b75ab9cb514a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/87498d8d9df44d829839b75ab9cb514a 2023-04-23 05:55:21,175 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/acb3991d58004197b238a56ec676c568 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/acb3991d58004197b238a56ec676c568 2023-04-23 05:55:21,176 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/031f33cc2ec0496cbfcf7ef43421ff4c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/031f33cc2ec0496cbfcf7ef43421ff4c 2023-04-23 05:55:21,177 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ab99a66694a14def97117c3d1f1c3e27 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ab99a66694a14def97117c3d1f1c3e27 2023-04-23 05:55:21,178 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/2f85da106031487faa506667dd269db0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/2f85da106031487faa506667dd269db0 2023-04-23 05:55:21,179 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/23cd8a6d424f4adfbeb0a1cc0f0c5117 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/23cd8a6d424f4adfbeb0a1cc0f0c5117 2023-04-23 05:55:21,180 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca60849574624bcf822bded65673c3a9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca60849574624bcf822bded65673c3a9 2023-04-23 05:55:21,180 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7b7fd4c83ad14515bdb442750a6bb6cf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7b7fd4c83ad14515bdb442750a6bb6cf 2023-04-23 05:55:21,181 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a0aa8f11cf484f37863b0f0334715fb4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a0aa8f11cf484f37863b0f0334715fb4 2023-04-23 05:55:21,182 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/51de38dfd08644eaaecadc56b0e0b49c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/51de38dfd08644eaaecadc56b0e0b49c 2023-04-23 05:55:21,183 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/1bcc6505af704cd0bfe4a3b2f88cc330 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/1bcc6505af704cd0bfe4a3b2f88cc330 2023-04-23 05:55:21,184 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/17ecbcceb2fc4705a7a5585bbdf0f6bc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/17ecbcceb2fc4705a7a5585bbdf0f6bc 2023-04-23 05:55:21,185 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9ebfacec72034f00af6e733b18404a50 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9ebfacec72034f00af6e733b18404a50 2023-04-23 05:55:21,186 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/410dfa6905934fafb240c5adfaf70203 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/410dfa6905934fafb240c5adfaf70203 2023-04-23 05:55:21,186 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/78c75f9332bf42be9778e1feecd641ca to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/78c75f9332bf42be9778e1feecd641ca 2023-04-23 05:55:21,187 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a5e72b1c54b44567acab6eb1d27b7140 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a5e72b1c54b44567acab6eb1d27b7140 2023-04-23 05:55:21,188 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0a17a6f7d58f47309acb2e6a97133597 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0a17a6f7d58f47309acb2e6a97133597 2023-04-23 05:55:21,189 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/eae45b335b83451ebe03783a81197d35 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/eae45b335b83451ebe03783a81197d35 2023-04-23 05:55:21,190 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/48a7d4352d8b43d8ad3ae684325882ba to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/48a7d4352d8b43d8ad3ae684325882ba 2023-04-23 05:55:21,190 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/725d83ae159647de92bb45ac770212d7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/725d83ae159647de92bb45ac770212d7 2023-04-23 05:55:21,191 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/79b8d45be0924d0e900044667de56b33 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/79b8d45be0924d0e900044667de56b33 2023-04-23 05:55:21,192 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca41f593090c4a2bbe0c4ced4c202b49 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/ca41f593090c4a2bbe0c4ced4c202b49 2023-04-23 05:55:21,193 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/f7c881349e054ae78d369f8916691b68 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/f7c881349e054ae78d369f8916691b68 2023-04-23 05:55:21,194 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/d1d3ec1ca6ca479ebe48a4eeaa09b23a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/d1d3ec1ca6ca479ebe48a4eeaa09b23a 2023-04-23 05:55:21,195 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9306f390719244b2afe2732e8208b305 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/9306f390719244b2afe2732e8208b305 2023-04-23 05:55:21,196 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a4930698f49d4c98a8a8f419e7cd73e4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a4930698f49d4c98a8a8f419e7cd73e4 2023-04-23 05:55:21,197 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0ed31b6e0c1b494a9bd602a18d60ee3a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0ed31b6e0c1b494a9bd602a18d60ee3a 2023-04-23 05:55:21,198 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7a5989d95fe24349b394c3cae2a2f279 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/7a5989d95fe24349b394c3cae2a2f279 2023-04-23 05:55:21,199 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a102360294644ee9ae054feb9c4bb23d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/a102360294644ee9ae054feb9c4bb23d 2023-04-23 05:55:21,200 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0b61c7cb08c949f1937338911140434a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/0b61c7cb08c949f1937338911140434a 2023-04-23 05:55:21,201 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/163586716e624e09ae0ed07ebdd26357 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/163586716e624e09ae0ed07ebdd26357 2023-04-23 05:55:21,202 DEBUG [StoreCloser-TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/72cc03dac4c04ba5ac186d88ec0c30ea to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/72cc03dac4c04ba5ac186d88ec0c30ea 2023-04-23 05:55:21,205 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/recovered.edits/1015.seqid, newMaxSeqId=1015, maxSeqId=1 2023-04-23 05:55:21,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd. 2023-04-23 05:55:21,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 911c079a878d9ede8370cba6d74e3efd: 2023-04-23 05:55:21,208 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:21,208 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=911c079a878d9ede8370cba6d74e3efd, regionState=CLOSED 2023-04-23 05:55:21,209 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229321208"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229321208"}]},"ts":"1682229321208"} 2023-04-23 05:55:21,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-23 05:55:21,210 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 911c079a878d9ede8370cba6d74e3efd, server=jenkins-hbase4.apache.org,40995,1682229223026 in 1.5160 sec 2023-04-23 05:55:21,211 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-23 05:55:21,211 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=911c079a878d9ede8370cba6d74e3efd, UNASSIGN in 1.5190 sec 2023-04-23 05:55:21,212 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229321212"}]},"ts":"1682229321212"} 2023-04-23 05:55:21,213 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-23 05:55:21,214 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-23 05:55:21,215 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5300 sec 2023-04-23 05:55:21,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-23 05:55:21,792 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-23 05:55:21,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-23 05:55:21,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:21,795 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:21,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-23 05:55:21,796 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:21,801 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:21,803 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/recovered.edits] 2023-04-23 05:55:21,805 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/7873b662248743e0a23b80a6dc0ea814 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/7873b662248743e0a23b80a6dc0ea814 2023-04-23 05:55:21,807 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c00bc4d6764c445d8d4abf5699294949 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/A/c00bc4d6764c445d8d4abf5699294949 2023-04-23 05:55:21,810 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/86f831720a304050bb65e78d232dc8c6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/86f831720a304050bb65e78d232dc8c6 2023-04-23 05:55:21,811 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/b1fe1256be5749df9403e755a5405316 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/B/b1fe1256be5749df9403e755a5405316 2023-04-23 05:55:21,814 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/99f06a321bb44bcd806ca84f34b266bd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/99f06a321bb44bcd806ca84f34b266bd 2023-04-23 05:55:21,815 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/beb7fc57e4e945bd9136b70e20150a18 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/C/beb7fc57e4e945bd9136b70e20150a18 2023-04-23 05:55:21,817 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/recovered.edits/1015.seqid to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd/recovered.edits/1015.seqid 2023-04-23 05:55:21,818 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/911c079a878d9ede8370cba6d74e3efd 2023-04-23 05:55:21,818 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:55:21,820 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:21,821 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-23 05:55:21,823 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-23 05:55:21,823 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:21,823 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-23 05:55:21,824 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682229321824"}]},"ts":"9223372036854775807"} 2023-04-23 05:55:21,825 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-23 05:55:21,825 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 911c079a878d9ede8370cba6d74e3efd, NAME => 'TestAcidGuarantees,,1682229293460.911c079a878d9ede8370cba6d74e3efd.', STARTKEY => '', ENDKEY => ''}] 2023-04-23 05:55:21,825 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-23 05:55:21,825 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682229321825"}]},"ts":"9223372036854775807"} 2023-04-23 05:55:21,826 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-23 05:55:21,828 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:21,829 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 35 msec 2023-04-23 05:55:21,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-23 05:55:21,897 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-23 05:55:21,909 INFO [Listener at localhost/41845] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=529 (was 541), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=350 (was 366), ProcessCount=169 (was 169), AvailableMemoryMB=12337 (was 12381) 2023-04-23 05:55:21,920 INFO [Listener at localhost/41845] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226, OpenFileDescriptor=529, MaxFileDescriptor=60000, SystemLoadAverage=350, ProcessCount=169, AvailableMemoryMB=12336 2023-04-23 05:55:21,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-23 05:55:21,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:55:21,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:21,925 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-23 05:55:21,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-23 05:55:21,925 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-23 05:55:21,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-23 05:55:21,926 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:21,927 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2 empty. 2023-04-23 05:55:21,927 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:21,927 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:55:22,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-23 05:55:22,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-23 05:55:22,335 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-23 05:55:22,337 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9b5a61538d9e2f8b71e356a41d6c31f2, NAME => 'TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp 2023-04-23 05:55:22,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-23 05:55:22,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:55:22,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9b5a61538d9e2f8b71e356a41d6c31f2, disabling compactions & flushes 2023-04-23 05:55:22,743 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:22,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:22,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. after waiting 0 ms 2023-04-23 05:55:22,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:22,743 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:22,743 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:22,746 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-23 05:55:22,747 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229322746"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229322746"}]},"ts":"1682229322746"} 2023-04-23 05:55:22,747 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-23 05:55:22,748 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-23 05:55:22,748 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229322748"}]},"ts":"1682229322748"} 2023-04-23 05:55:22,749 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-23 05:55:22,753 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, ASSIGN}] 2023-04-23 05:55:22,754 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, ASSIGN 2023-04-23 05:55:22,755 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=false 2023-04-23 05:55:22,906 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=9b5a61538d9e2f8b71e356a41d6c31f2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:22,906 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229322906"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229322906"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229322906"}]},"ts":"1682229322906"} 2023-04-23 05:55:22,907 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:55:23,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-23 05:55:23,063 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:23,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9b5a61538d9e2f8b71e356a41d6c31f2, NAME => 'TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:55:23,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:23,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:55:23,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:23,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:23,064 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:23,065 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A 2023-04-23 05:55:23,065 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A 2023-04-23 05:55:23,066 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:55:23,066 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9b5a61538d9e2f8b71e356a41d6c31f2 columnFamilyName A 2023-04-23 05:55:23,066 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(310): Store=9b5a61538d9e2f8b71e356a41d6c31f2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:55:23,066 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:23,067 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B 2023-04-23 05:55:23,067 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B 2023-04-23 05:55:23,067 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:55:23,068 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9b5a61538d9e2f8b71e356a41d6c31f2 columnFamilyName B 2023-04-23 05:55:23,068 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(310): Store=9b5a61538d9e2f8b71e356a41d6c31f2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:55:23,068 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:23,069 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C 2023-04-23 05:55:23,069 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C 2023-04-23 05:55:23,069 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:55:23,069 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9b5a61538d9e2f8b71e356a41d6c31f2 columnFamilyName C 2023-04-23 05:55:23,069 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(310): Store=9b5a61538d9e2f8b71e356a41d6c31f2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:55:23,070 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:23,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:23,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:23,072 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:55:23,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:23,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-23 05:55:23,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9b5a61538d9e2f8b71e356a41d6c31f2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73847697, jitterRate=0.10041643679141998}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:55:23,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:23,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., pid=37, masterSystemTime=1682229323059 2023-04-23 05:55:23,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:23,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:23,077 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=9b5a61538d9e2f8b71e356a41d6c31f2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:23,077 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229323077"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229323077"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229323077"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229323077"}]},"ts":"1682229323077"} 2023-04-23 05:55:23,079 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-23 05:55:23,079 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 in 171 msec 2023-04-23 05:55:23,080 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-23 05:55:23,080 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, ASSIGN in 326 msec 2023-04-23 05:55:23,080 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-23 05:55:23,081 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229323080"}]},"ts":"1682229323080"} 2023-04-23 05:55:23,081 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-23 05:55:23,083 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-23 05:55:23,084 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-23 05:55:24,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-23 05:55:24,031 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-23 05:55:24,032 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x67b6d133 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,038 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@494529f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,039 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:24,042 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57264, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:24,043 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-23 05:55:24,044 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56934, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-23 05:55:24,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-23 05:55:24,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:55:24,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:24,459 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-23 05:55:24,460 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-23 05:55:24,460 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-23 05:55:24,462 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-23 05:55:24,465 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, REOPEN/MOVE}] 2023-04-23 05:55:24,465 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, REOPEN/MOVE 2023-04-23 05:55:24,466 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9b5a61538d9e2f8b71e356a41d6c31f2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:24,466 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229324466"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229324466"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229324466"}]},"ts":"1682229324466"} 2023-04-23 05:55:24,467 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:55:24,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9b5a61538d9e2f8b71e356a41d6c31f2, disabling compactions & flushes 2023-04-23 05:55:24,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:24,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:24,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. after waiting 0 ms 2023-04-23 05:55:24,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:24,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-23 05:55:24,623 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:24,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:24,623 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 9b5a61538d9e2f8b71e356a41d6c31f2 to self. 2023-04-23 05:55:24,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,625 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9b5a61538d9e2f8b71e356a41d6c31f2, regionState=CLOSED 2023-04-23 05:55:24,625 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229324625"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229324625"}]},"ts":"1682229324625"} 2023-04-23 05:55:24,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-23 05:55:24,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 in 159 msec 2023-04-23 05:55:24,627 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=true 2023-04-23 05:55:24,778 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9b5a61538d9e2f8b71e356a41d6c31f2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:24,778 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229324778"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229324778"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229324778"}]},"ts":"1682229324778"} 2023-04-23 05:55:24,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:55:24,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:24,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9b5a61538d9e2f8b71e356a41d6c31f2, NAME => 'TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:55:24,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:55:24,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,936 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,936 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A 2023-04-23 05:55:24,937 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A 2023-04-23 05:55:24,937 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:55:24,937 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9b5a61538d9e2f8b71e356a41d6c31f2 columnFamilyName A 2023-04-23 05:55:24,938 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(310): Store=9b5a61538d9e2f8b71e356a41d6c31f2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:55:24,939 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,939 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B 2023-04-23 05:55:24,939 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B 2023-04-23 05:55:24,939 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:55:24,940 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9b5a61538d9e2f8b71e356a41d6c31f2 columnFamilyName B 2023-04-23 05:55:24,940 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(310): Store=9b5a61538d9e2f8b71e356a41d6c31f2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:55:24,940 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,941 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C 2023-04-23 05:55:24,941 DEBUG [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C 2023-04-23 05:55:24,941 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:55:24,941 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9b5a61538d9e2f8b71e356a41d6c31f2 columnFamilyName C 2023-04-23 05:55:24,941 INFO [StoreOpener-9b5a61538d9e2f8b71e356a41d6c31f2-1] regionserver.HStore(310): Store=9b5a61538d9e2f8b71e356a41d6c31f2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:55:24,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:24,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:55:24,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:24,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9b5a61538d9e2f8b71e356a41d6c31f2; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71403418, jitterRate=0.06399384140968323}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:55:24,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:24,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., pid=42, masterSystemTime=1682229324931 2023-04-23 05:55:24,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:24,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:24,949 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=9b5a61538d9e2f8b71e356a41d6c31f2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:24,949 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229324948"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229324948"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229324948"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229324948"}]},"ts":"1682229324948"} 2023-04-23 05:55:24,951 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-23 05:55:24,951 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 in 170 msec 2023-04-23 05:55:24,952 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-23 05:55:24,952 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, REOPEN/MOVE in 486 msec 2023-04-23 05:55:24,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-23 05:55:24,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-23 05:55:24,955 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 907 msec 2023-04-23 05:55:24,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-23 05:55:24,957 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x1cee66b5 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,961 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d800352, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,961 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x4c1d19b3 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,965 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2329054b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,965 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x14e151fb to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,971 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30951255, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,971 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x455a88b1 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,975 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5baa6768, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,975 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x722b2407 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,979 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63a535f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,979 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x2d4bfeca to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,983 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@678b4c3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,983 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x313c7ad6 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,987 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43736f95, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,988 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x20d28b68 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,994 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72eaaf6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,994 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x750d2a7d to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:24,998 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4130667f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:24,998 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x27313b38 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:25,002 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b73a361, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:25,007 DEBUG [hconnection-0x15a4ecd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,007 DEBUG [hconnection-0x71fa59ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,007 DEBUG [hconnection-0x2017cb4f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,009 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57272, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:25,010 DEBUG [hconnection-0x34211721-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,010 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,011 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57282, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,012 DEBUG [hconnection-0x2a4a00dd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,012 DEBUG [hconnection-0x734984b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,014 DEBUG [hconnection-0x40a2bd9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,015 DEBUG [hconnection-0x1002c3aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:25,015 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57292, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,015 DEBUG [hconnection-0x1047fcaa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,015 DEBUG [hconnection-0x479f5e24-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:25,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:25,018 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:25,018 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,018 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:25,018 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:25,018 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57314, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,020 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,020 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,020 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,022 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,024 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,021 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57298, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,026 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:25,029 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:25,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:25,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:25,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:25,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:25,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:55:25,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:25,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:25,041 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,041 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:25,041 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:25,041 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:25,041 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:25,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:25,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:25,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:25,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:25,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:25,044 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:25,044 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:25,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41360ac0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:25,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:25,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:25,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,055 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:25,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:25,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,056 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:25,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:25,060 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:25,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:25,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:25,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:25,060 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:25,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229385061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229385062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229385062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229385068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,069 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,071 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229385070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,074 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,074 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,074 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,074 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:25,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:25,074 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,075 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:25,075 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:25,077 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423c55d34acf71841e7b376f01e542f7470 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c55d34acf71841e7b376f01e542f7470 2023-04-23 05:55:25,092 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/605492c5dd914172b39f085ae4140fac 2023-04-23 05:55:25,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229385169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229385169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229385169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229385169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,175 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:25,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:25,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:25,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229385175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:25,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:25,183 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:25,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:25,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:25,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,185 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:25,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:25,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:25,191 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,191 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:25,191 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:25,191 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@746889e6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:25,195 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:25,195 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,196 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:25,197 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,197 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:25,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:25,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:25,202 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:25,202 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:25,202 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:25,202 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:25,202 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:25,203 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,203 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,204 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,204 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,209 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:25,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:25,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:25,210 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:25,210 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:25,310 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:25,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:25,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:25,314 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:25,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:25,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:25,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,315 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,316 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,316 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,316 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:25,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:25,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:25,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:25,320 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,320 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:25,320 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:25,320 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:25,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:25,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:25,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:25,323 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,323 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,323 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,323 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@617ec98c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:25,325 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:25,325 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,326 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:25,326 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:25,326 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:25,326 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,326 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:25,327 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,327 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,327 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,328 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:25,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:25,330 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:25,330 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:25,330 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:25,330 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:25,330 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:25,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,332 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,336 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,336 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,336 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:25,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:25,336 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,337 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,336 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,337 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,337 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,337 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,337 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:25,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:25,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,337 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:25,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229385373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229385374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229385374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229385376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229385379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,437 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:25,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:25,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:25,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:25,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:25,441 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:25,441 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,441 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:25,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:25,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,443 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,443 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,444 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,444 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:25,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:25,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:25,445 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,445 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:25,445 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:25,445 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:25,445 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:25,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,447 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,447 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:25,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0072a2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:25,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,450 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,450 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,450 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:25,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:25,457 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:25,459 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:25,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:25,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:25,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:25,459 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:25,460 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,460 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,460 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,461 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,462 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,464 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,464 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,464 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,465 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,465 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:25,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:25,465 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,465 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,465 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,465 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,465 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,465 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:25,465 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:25,506 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/e3d7687c82fb45819ad732fdbf6e8287 2023-04-23 05:55:25,565 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:25,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:25,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:25,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:25,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:25,570 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:25,570 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,570 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:25,570 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:25,570 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,570 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:25,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:25,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:25,576 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,576 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:25,576 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:25,576 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:25,576 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:25,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b911b28[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:25,580 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:25,580 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,581 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,581 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:25,582 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:25,583 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:25,585 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:25,585 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:25,585 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:25,585 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:25,585 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:25,588 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,588 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,588 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,589 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,591 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,593 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:25,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,593 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,593 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:25,594 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:25,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229385677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229385678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229385678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229385679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:25,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229385687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,695 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:25,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:25,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:25,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:25,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:25,698 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:25,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:25,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:25,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,700 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,700 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,700 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,700 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:25,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:25,701 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:25,701 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,701 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:25,703 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:25,703 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:25,704 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,704 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:25,704 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:25,704 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:25,706 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,706 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,706 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:25,711 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b069805[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:25,713 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:25,713 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,715 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:25,715 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:25,715 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,715 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:25,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:25,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,717 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,717 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:25,719 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:25,719 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:25,719 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:25,719 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:25,719 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:25,720 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,720 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,720 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,721 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,722 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,724 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,724 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,724 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,724 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,724 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,728 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,728 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,728 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:25,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:25,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:25,728 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,728 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,728 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:25,728 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:25,828 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:25,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:25,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:25,832 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:25,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:25,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:25,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,833 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,835 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,835 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,836 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,836 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:25,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:25,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:25,837 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:25,837 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,837 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:25,838 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,838 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:25,838 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:25,838 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:25,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:25,840 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,840 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,840 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,840 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c230840[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:25,841 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:25,842 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,843 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,843 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:25,843 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,843 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:25,843 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:25,843 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:25,844 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:25,844 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,844 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:25,845 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:25,847 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:25,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:25,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:25,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:25,847 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:25,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:25,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:25,851 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,852 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:25,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:25,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:25,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:25,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:25,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:25,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:25,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:25,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:25,858 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:25,859 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:25,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/a70fe58f320e490c80153a82ea168430 2023-04-23 05:55:25,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/605492c5dd914172b39f085ae4140fac as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/605492c5dd914172b39f085ae4140fac 2023-04-23 05:55:25,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/605492c5dd914172b39f085ae4140fac, entries=150, sequenceid=17, filesize=25.1 K 2023-04-23 05:55:25,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/e3d7687c82fb45819ad732fdbf6e8287 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e3d7687c82fb45819ad732fdbf6e8287 2023-04-23 05:55:25,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e3d7687c82fb45819ad732fdbf6e8287, entries=150, sequenceid=17, filesize=11.6 K 2023-04-23 05:55:25,938 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/a70fe58f320e490c80153a82ea168430 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a70fe58f320e490c80153a82ea168430 2023-04-23 05:55:25,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a70fe58f320e490c80153a82ea168430, entries=150, sequenceid=17, filesize=11.6 K 2023-04-23 05:55:25,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 906ms, sequenceid=17, compaction requested=false 2023-04-23 05:55:25,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:25,959 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:25,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:25,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:25,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:25,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:25,963 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:25,963 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,963 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:25,963 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:25,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,965 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:25,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:25,965 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:25,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:25,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:25,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:25,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:25,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:25,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:25,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:25,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:25,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:25,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:25,967 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:25,967 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:25,967 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:55:25,967 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:25,967 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:25,968 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:25,968 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:25,968 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:25,968 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:26,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:26,186 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:26,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229386194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229386195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229386196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229386197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229386198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229386300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229386304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229386304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229386304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229386305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,383 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423c7f7e60a8e734dd09fd8078eb2c1682e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c7f7e60a8e734dd09fd8078eb2c1682e 2023-04-23 05:55:26,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229386506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229386509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229386509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229386512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229386513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,746 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-23 05:55:26,791 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/96f9317e2cf4436ba692c71e0ad702ac 2023-04-23 05:55:26,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229386811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229386814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229386815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229386817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:26,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:26,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229386817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,210 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/a4f9a50da80d46ff947c01e306df2999 2023-04-23 05:55:27,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:27,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229387316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:27,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229387321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:27,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229387323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:27,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229387323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:27,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229387325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,621 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/47a13dc1da484c1c9615159c3ae26138 2023-04-23 05:55:27,625 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/96f9317e2cf4436ba692c71e0ad702ac as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/96f9317e2cf4436ba692c71e0ad702ac 2023-04-23 05:55:27,629 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/96f9317e2cf4436ba692c71e0ad702ac, entries=150, sequenceid=37, filesize=25.1 K 2023-04-23 05:55:27,630 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/a4f9a50da80d46ff947c01e306df2999 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a4f9a50da80d46ff947c01e306df2999 2023-04-23 05:55:27,634 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a4f9a50da80d46ff947c01e306df2999, entries=150, sequenceid=37, filesize=11.6 K 2023-04-23 05:55:27,635 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/47a13dc1da484c1c9615159c3ae26138 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/47a13dc1da484c1c9615159c3ae26138 2023-04-23 05:55:27,639 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/47a13dc1da484c1c9615159c3ae26138, entries=150, sequenceid=37, filesize=11.6 K 2023-04-23 05:55:27,640 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 1673ms, sequenceid=37, compaction requested=false 2023-04-23 05:55:27,640 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:27,640 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:27,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:27,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:27,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:27,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:27,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:27,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:27,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:27,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55aff604[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:27,646 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:27,646 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:27,648 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:27,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:27,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:27,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:27,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:27,653 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:27,653 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:27,653 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:27,653 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:27,653 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:27,654 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:27,654 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:27,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:27,654 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:27,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:27,655 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,656 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:27,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:27,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:27,660 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:27,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:27,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:27,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:27,660 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:27,660 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,660 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:27,660 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:27,660 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:27,660 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:27,660 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:27,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:27,760 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:27,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:27,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:27,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:27,764 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:27,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:27,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:27,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:27,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:27,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:27,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:27,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:27,767 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:27,767 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:27,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:27,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:27,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:27,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:27,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:27,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:27,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:27,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:27,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:27,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:27,768 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:27,768 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:27,769 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:55:27,769 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:27,769 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:27,769 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:27,769 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:27,769 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:27,769 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:27,785 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423f8f4582a82904f8d80a4d50beae51e5f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f8f4582a82904f8d80a4d50beae51e5f 2023-04-23 05:55:28,193 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/a8113ec86a9848f8aacbac89cf9d11a7 2023-04-23 05:55:28,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:28,326 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:28,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229388345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229388346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229388348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229388349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229388352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229388453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229388453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229388455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229388455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229388455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,603 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/db58e93fa9fc456782ba487bddab2af2 2023-04-23 05:55:28,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229388654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229388658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229388663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229388664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229388664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229388960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229388961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229388970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229388971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:28,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:28,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229388974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,015 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/7ac04993f7d24e80b2a37539996833ef 2023-04-23 05:55:29,023 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/a8113ec86a9848f8aacbac89cf9d11a7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/a8113ec86a9848f8aacbac89cf9d11a7 2023-04-23 05:55:29,028 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/a8113ec86a9848f8aacbac89cf9d11a7, entries=150, sequenceid=49, filesize=25.1 K 2023-04-23 05:55:29,029 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/db58e93fa9fc456782ba487bddab2af2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/db58e93fa9fc456782ba487bddab2af2 2023-04-23 05:55:29,041 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/db58e93fa9fc456782ba487bddab2af2, entries=150, sequenceid=49, filesize=11.6 K 2023-04-23 05:55:29,047 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/7ac04993f7d24e80b2a37539996833ef as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7ac04993f7d24e80b2a37539996833ef 2023-04-23 05:55:29,056 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7ac04993f7d24e80b2a37539996833ef, entries=150, sequenceid=49, filesize=11.6 K 2023-04-23 05:55:29,057 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 1288ms, sequenceid=49, compaction requested=true 2023-04-23 05:55:29,057 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:29,057 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:29,060 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3684efc2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:29,066 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:29,066 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:29,067 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:29,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:29,068 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,068 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:29,069 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:29,069 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:29,069 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:29,070 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:29,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:29,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:29,074 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:29,074 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:29,074 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:29,074 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:29,074 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:29,076 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:29,076 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:29,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:29,076 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:29,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:29,077 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,079 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:29,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:29,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:29,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:29,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:29,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:29,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:29,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:29,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:29,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:29,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:29,081 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:29,081 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:29,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:29,182 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:29,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:29,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:29,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:29,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:29,185 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:29,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:29,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:29,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:29,186 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:29,186 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:29,187 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:29,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:29,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:29,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:29,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:29,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:29,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:29,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:29,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:29,189 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:29,189 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:29,189 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:55:29,189 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:29,189 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:29,189 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:29,189 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:29,189 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:29,189 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:29,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:29,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:29,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229389484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229389484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229389486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229389487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229389494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229389595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229389595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229389596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,608 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304234c2e0ef293334e9095da8523356206b2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304234c2e0ef293334e9095da8523356206b2 2023-04-23 05:55:29,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229389602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229389798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229389799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229389802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:29,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:29,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229389809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,024 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/2f1942149f7a42e7a215cb671ab09ce6 2023-04-23 05:55:30,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:30,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229390106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229390106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229390108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:30,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229390113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,435 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/e0a922bb658f441cac776f4bb979488c 2023-04-23 05:55:30,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:30,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229390497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:30,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229390613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:30,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229390613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:30,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229390615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:30,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229390622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,850 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/75896ffdb3b34ad3b0c0a32c796acbf8 2023-04-23 05:55:30,855 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/2f1942149f7a42e7a215cb671ab09ce6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/2f1942149f7a42e7a215cb671ab09ce6 2023-04-23 05:55:30,859 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/2f1942149f7a42e7a215cb671ab09ce6, entries=150, sequenceid=69, filesize=25.1 K 2023-04-23 05:55:30,860 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/e0a922bb658f441cac776f4bb979488c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e0a922bb658f441cac776f4bb979488c 2023-04-23 05:55:30,864 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e0a922bb658f441cac776f4bb979488c, entries=150, sequenceid=69, filesize=11.6 K 2023-04-23 05:55:30,865 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/75896ffdb3b34ad3b0c0a32c796acbf8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/75896ffdb3b34ad3b0c0a32c796acbf8 2023-04-23 05:55:30,870 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/75896ffdb3b34ad3b0c0a32c796acbf8, entries=150, sequenceid=69, filesize=11.6 K 2023-04-23 05:55:30,870 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 1681ms, sequenceid=69, compaction requested=true 2023-04-23 05:55:30,871 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:30,871 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:30,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:30,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:30,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:30,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:30,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:30,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:30,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:30,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@592d9fa7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:30,874 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:30,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:30,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:30,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:30,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:30,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:30,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:30,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:30,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:30,877 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:30,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:30,879 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:30,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:30,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:30,880 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:30,880 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:30,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:30,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:30,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:30,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:30,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:30,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:30,885 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,890 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:30,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:30,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:30,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:30,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:30,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:30,891 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:30,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,891 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:30,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:30,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:30,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:30,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:30,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:30,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:30,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:30,991 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:30,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:30,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:30,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:30,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:30,994 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:30,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:30,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:30,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:30,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:30,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:30,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:30,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:30,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:30,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:30,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:30,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:30,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:30,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:30,998 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:30,998 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:30,998 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:55:30,998 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:30,998 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:30,999 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:30,999 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:30,999 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:30,999 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:31,017 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042344bf2a05b8d4439b993cfafac6c9fe6f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042344bf2a05b8d4439b993cfafac6c9fe6f 2023-04-23 05:55:31,426 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=81, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/df21cae0d61d40f48a8054302a7ae44a 2023-04-23 05:55:31,436 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/96b1e64c73454bec830961c786296315 2023-04-23 05:55:31,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:31,632 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:31,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:31,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:31,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229391659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229391659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:31,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229391659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:31,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229391663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:31,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229391766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:31,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229391767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:31,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229391767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:31,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229391767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,846 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/c303dc021f9c4608961733237d0c60c1 2023-04-23 05:55:31,850 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/df21cae0d61d40f48a8054302a7ae44a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/df21cae0d61d40f48a8054302a7ae44a 2023-04-23 05:55:31,853 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/df21cae0d61d40f48a8054302a7ae44a, entries=150, sequenceid=81, filesize=25.1 K 2023-04-23 05:55:31,854 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/96b1e64c73454bec830961c786296315 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/96b1e64c73454bec830961c786296315 2023-04-23 05:55:31,857 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/96b1e64c73454bec830961c786296315, entries=150, sequenceid=81, filesize=11.6 K 2023-04-23 05:55:31,858 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/c303dc021f9c4608961733237d0c60c1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c303dc021f9c4608961733237d0c60c1 2023-04-23 05:55:31,861 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c303dc021f9c4608961733237d0c60c1, entries=150, sequenceid=81, filesize=11.6 K 2023-04-23 05:55:31,862 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 864ms, sequenceid=81, compaction requested=true 2023-04-23 05:55:31,862 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:31,862 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:31,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:31,863 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:31,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d18fdca[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:31,866 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:31,866 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,868 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,869 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:31,869 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:31,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:31,869 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:31,870 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:31,873 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:31,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:31,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:31,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:31,873 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:31,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:31,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:31,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:31,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:31,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:31,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:31,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:31,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:31,880 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:31,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:31,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:31,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:31,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:31,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:31,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:31,881 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:31,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:31,881 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:31,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:31,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:31,974 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-23 05:55:31,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:31,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:31,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229331657, max timestamp=1682229331973Num uniques 859; 2023-04-23 05:55:31,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:31,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:31,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:31,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229331657, max timestamp=1682229331973Num uniques 859; 2023-04-23 05:55:31,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:31,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:31,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:31,981 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:31,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:31,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:31,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:31,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:31,985 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:31,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:31,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:31,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:31,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:31,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:31,988 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:31,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:31,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:31,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:31,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:31,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:31,990 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:31,990 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:31,990 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:31,990 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:31,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:31,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:31,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272d35bb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:31,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:31,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:31,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:31,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:31,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:31,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:31,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:31,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:31,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:31,998 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:31,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:31,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:31,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:31,999 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:32,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:32,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:32,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:32,005 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,005 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:32,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,005 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:32,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229332009, max timestamp=1682229332018Num uniques 859; 2023-04-23 05:55:32,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:32,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229332009, max timestamp=1682229332018Num uniques 859; 2023-04-23 05:55:32,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:32,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229332009, max timestamp=1682229332018Num uniques 859; 2023-04-23 05:55:32,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:32,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229392054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229392054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229392063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229392064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,105 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:32,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:32,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:32,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:32,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:32,109 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:32,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:32,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:32,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,110 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,110 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:32,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:32,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:32,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:32,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:32,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:32,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,114 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,114 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,114 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:32,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36c63463[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:32,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:32,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,118 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:32,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:32,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:32,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:32,120 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:32,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:32,122 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:32,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:32,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:32,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:32,122 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:32,123 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,123 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,125 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:32,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:32,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:32,128 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,129 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:32,129 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:32,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229392164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229392164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229392173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229392173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,229 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:32,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:32,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:32,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:32,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:32,233 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:32,233 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,233 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:32,233 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:32,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:32,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:32,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:32,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:32,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:32,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:32,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:32,241 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,241 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:32,241 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:32,241 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:32,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:32,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:32,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:32,242 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,243 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,243 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32345bb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:32,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:32,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:32,246 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:32,246 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:32,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:32,248 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:32,251 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:32,251 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:32,251 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:32,251 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:32,251 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:32,252 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,252 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,252 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,253 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,255 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:32,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:32,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,257 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,257 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:32,257 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:32,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,357 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:32,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:32,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:32,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:32,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:32,364 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:32,364 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,364 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:32,364 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:32,365 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,365 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,366 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:32,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:32,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:32,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:32,368 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,368 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:32,368 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:32,368 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:32,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229392370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229392370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@491f1eb8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:32,374 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:32,374 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,375 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:32,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:32,376 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,376 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:32,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:32,377 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,377 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:32,377 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,377 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:32,379 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:32,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:32,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:32,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:32,379 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:32,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229392380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229392380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,383 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,384 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,387 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:32,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,388 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,388 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:32,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,388 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:32,388 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:32,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,390 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423b67a04f691084f318e07a8d19610ac27 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423b67a04f691084f318e07a8d19610ac27 2023-04-23 05:55:32,488 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:32,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:32,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:32,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:32,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:32,492 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:32,492 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,492 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:32,492 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:32,492 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,493 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,494 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,494 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,494 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,494 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:32,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:32,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:32,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:32,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:32,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:32,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:32,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:32,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:32,495 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,495 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:32,496 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:32,496 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:32,496 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:32,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:32,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d16acd[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:32,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:32,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,501 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:32,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:32,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:32,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:32,504 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,504 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:32,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:32,506 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:32,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:32,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:32,507 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:32,507 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:32,508 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,508 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,508 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,509 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,510 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:32,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:32,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,514 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,515 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,515 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,515 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,515 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:32,515 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:32,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229392519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,528 DEBUG [Thread-2250] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:55:32,615 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:32,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:32,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:32,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:32,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:32,618 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:32,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:32,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:32,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,620 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,620 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,620 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,620 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:32,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:32,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:32,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:32,621 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:32,621 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,621 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:32,622 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,622 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:32,622 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:32,622 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:32,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:32,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b33378[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:32,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:32,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:32,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:32,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:32,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:32,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:32,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:32,634 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:32,634 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:32,634 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:32,634 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:32,635 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:32,637 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,637 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,637 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,638 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,639 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:32,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:32,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,641 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,641 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:32,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,641 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:32,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229392673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229392675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229392683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:32,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229392686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,742 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:32,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:32,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:32,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:32,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:32,746 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:32,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:32,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:32,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,747 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:32,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:32,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:32,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:32,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:32,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:32,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:32,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:32,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:32,753 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:32,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19a4b048[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:32,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:32,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,756 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:32,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:32,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:32,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:32,758 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,758 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:32,758 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,758 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:32,761 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:32,761 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:32,761 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:32,761 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:32,761 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:32,763 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,763 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,763 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,765 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:32,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:32,769 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:32,769 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:32,796 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=102, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/74304af6d6f84ff9b713def30718e10d 2023-04-23 05:55:32,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/e9e4cc4971d84c5b90fe3953031f5d82 2023-04-23 05:55:32,869 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:32,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:32,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:32,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:32,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:32,873 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:32,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:32,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:32,874 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,874 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:32,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:32,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:32,901 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:32,901 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:32,901 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:32,912 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:32,912 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:32,912 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:32,912 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:32,912 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:32,916 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:32,918 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,918 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,919 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,919 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:32,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:32,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@287f971a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:32,921 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:32,921 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,922 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:32,923 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,923 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:32,924 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,924 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:32,924 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,924 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/ac9d1aaeb71f44cb86c2d79e12db87bc 2023-04-23 05:55:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/74304af6d6f84ff9b713def30718e10d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/74304af6d6f84ff9b713def30718e10d 2023-04-23 05:55:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:32,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:32,931 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:32,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:32,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:32,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:32,931 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:32,935 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:32,935 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,935 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,937 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:32,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/74304af6d6f84ff9b713def30718e10d, entries=200, sequenceid=102, filesize=31.9 K 2023-04-23 05:55:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/e9e4cc4971d84c5b90fe3953031f5d82 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e9e4cc4971d84c5b90fe3953031f5d82 2023-04-23 05:55:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:32,940 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:32,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e9e4cc4971d84c5b90fe3953031f5d82, entries=150, sequenceid=102, filesize=11.6 K 2023-04-23 05:55:32,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:32,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:32,948 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:32,949 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:32,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:32,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:32,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/ac9d1aaeb71f44cb86c2d79e12db87bc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/ac9d1aaeb71f44cb86c2d79e12db87bc 2023-04-23 05:55:32,949 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:32,949 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:32,952 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:32,952 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:32,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:32,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:32,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/ac9d1aaeb71f44cb86c2d79e12db87bc, entries=150, sequenceid=102, filesize=11.6 K 2023-04-23 05:55:32,956 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~58.14 KB/59540, heapSize ~164.13 KB/168064, currentSize=134.18 KB/137400 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 981ms, sequenceid=102, compaction requested=true 2023-04-23 05:55:32,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:32,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:32,956 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:55:32,956 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:55:32,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:32,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:32,958 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:55:32,958 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 161416 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:55:32,958 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/B is initiating minor compaction (all files) 2023-04-23 05:55:32,958 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/A is initiating minor compaction (all files) 2023-04-23 05:55:32,958 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/B in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,958 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/A in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:32,958 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e3d7687c82fb45819ad732fdbf6e8287, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a4f9a50da80d46ff947c01e306df2999, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/db58e93fa9fc456782ba487bddab2af2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e0a922bb658f441cac776f4bb979488c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/96b1e64c73454bec830961c786296315, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e9e4cc4971d84c5b90fe3953031f5d82] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=69.6 K 2023-04-23 05:55:32,958 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/605492c5dd914172b39f085ae4140fac, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/96f9317e2cf4436ba692c71e0ad702ac, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/a8113ec86a9848f8aacbac89cf9d11a7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/2f1942149f7a42e7a215cb671ab09ce6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/df21cae0d61d40f48a8054302a7ae44a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/74304af6d6f84ff9b713def30718e10d] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=157.6 K 2023-04-23 05:55:32,958 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e3d7687c82fb45819ad732fdbf6e8287, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682229325030 2023-04-23 05:55:32,959 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 605492c5dd914172b39f085ae4140fac, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682229325030 2023-04-23 05:55:32,959 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a4f9a50da80d46ff947c01e306df2999, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1682229325059 2023-04-23 05:55:32,959 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 96f9317e2cf4436ba692c71e0ad702ac, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1682229325059 2023-04-23 05:55:32,959 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting db58e93fa9fc456782ba487bddab2af2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682229326191 2023-04-23 05:55:32,959 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a8113ec86a9848f8aacbac89cf9d11a7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682229326191 2023-04-23 05:55:32,959 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e0a922bb658f441cac776f4bb979488c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682229328343 2023-04-23 05:55:32,960 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 2f1942149f7a42e7a215cb671ab09ce6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682229328343 2023-04-23 05:55:32,960 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 96b1e64c73454bec830961c786296315, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682229329480 2023-04-23 05:55:32,960 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting df21cae0d61d40f48a8054302a7ae44a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682229329480 2023-04-23 05:55:32,960 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e9e4cc4971d84c5b90fe3953031f5d82, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682229331657 2023-04-23 05:55:32,965 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 74304af6d6f84ff9b713def30718e10d, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682229331657 2023-04-23 05:55:32,991 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#B#compaction#314 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:33,003 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#A#compaction#313 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:33,036 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/171576687bb5460b88479d976c078435 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/171576687bb5460b88479d976c078435 2023-04-23 05:55:33,042 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/B of 9b5a61538d9e2f8b71e356a41d6c31f2 into 171576687bb5460b88479d976c078435(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:33,042 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:33,042 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/B, priority=10, startTime=1682229332956; duration=0sec 2023-04-23 05:55:33,042 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:33,042 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:55:33,048 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:55:33,048 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/C is initiating minor compaction (all files) 2023-04-23 05:55:33,048 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/C in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:33,048 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a70fe58f320e490c80153a82ea168430, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/47a13dc1da484c1c9615159c3ae26138, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7ac04993f7d24e80b2a37539996833ef, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/75896ffdb3b34ad3b0c0a32c796acbf8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c303dc021f9c4608961733237d0c60c1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/ac9d1aaeb71f44cb86c2d79e12db87bc] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=69.6 K 2023-04-23 05:55:33,049 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a70fe58f320e490c80153a82ea168430, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1682229325030 2023-04-23 05:55:33,049 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 47a13dc1da484c1c9615159c3ae26138, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1682229325059 2023-04-23 05:55:33,049 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:33,050 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7ac04993f7d24e80b2a37539996833ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682229326191 2023-04-23 05:55:33,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:33,050 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 75896ffdb3b34ad3b0c0a32c796acbf8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682229328343 2023-04-23 05:55:33,051 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c303dc021f9c4608961733237d0c60c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682229329480 2023-04-23 05:55:33,052 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ac9d1aaeb71f44cb86c2d79e12db87bc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682229331657 2023-04-23 05:55:33,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:33,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:33,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:33,072 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:33,072 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:33,072 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:33,072 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:33,073 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:33,073 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:33,074 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,076 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,076 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:33,076 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:33,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:33,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:33,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:33,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:33,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:33,082 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#C#compaction#315 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:33,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:33,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:33,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:33,089 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:33,089 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:33,089 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:33,089 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:33,089 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-23 05:55:33,089 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:33,089 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:33,089 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:33,090 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:33,090 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:33,090 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:33,145 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423a6a7f6f2a6d845e28b0331d74f024a6b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a6a7f6f2a6d845e28b0331d74f024a6b 2023-04-23 05:55:33,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:33,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:33,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229393209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229393211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229393211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229393212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229393317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229393322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229393322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229393322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,510 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/7d278b255af04de5990e72a626d4c4dc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7d278b255af04de5990e72a626d4c4dc 2023-04-23 05:55:33,514 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/C of 9b5a61538d9e2f8b71e356a41d6c31f2 into 7d278b255af04de5990e72a626d4c4dc(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:33,514 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:33,514 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/C, priority=10, startTime=1682229332956; duration=0sec 2023-04-23 05:55:33,514 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:33,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229393527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229393534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229393535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229393535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,570 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=140, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/3e25b8903be5475eb9ef7feff43cb9e8 2023-04-23 05:55:33,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229393834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229393843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229393844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:33,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229393845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:33,884 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/610e0f9636ce4568a5d1ca5ccf1eaffe as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/610e0f9636ce4568a5d1ca5ccf1eaffe 2023-04-23 05:55:33,900 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/A of 9b5a61538d9e2f8b71e356a41d6c31f2 into 610e0f9636ce4568a5d1ca5ccf1eaffe(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:33,900 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:33,900 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/A, priority=10, startTime=1682229332956; duration=0sec 2023-04-23 05:55:33,900 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:33,997 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/0a1f7a2e98f642c0bac8bc18e23f54a9 2023-04-23 05:55:34,007 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/652182f73a9e457d9df6371dbffd9817 2023-04-23 05:55:34,011 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/3e25b8903be5475eb9ef7feff43cb9e8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3e25b8903be5475eb9ef7feff43cb9e8 2023-04-23 05:55:34,015 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3e25b8903be5475eb9ef7feff43cb9e8, entries=150, sequenceid=140, filesize=25.3 K 2023-04-23 05:55:34,016 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/0a1f7a2e98f642c0bac8bc18e23f54a9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/0a1f7a2e98f642c0bac8bc18e23f54a9 2023-04-23 05:55:34,023 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/0a1f7a2e98f642c0bac8bc18e23f54a9, entries=150, sequenceid=140, filesize=11.8 K 2023-04-23 05:55:34,024 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/652182f73a9e457d9df6371dbffd9817 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/652182f73a9e457d9df6371dbffd9817 2023-04-23 05:55:34,029 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/652182f73a9e457d9df6371dbffd9817, entries=150, sequenceid=140, filesize=11.8 K 2023-04-23 05:55:34,029 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 940ms, sequenceid=140, compaction requested=false 2023-04-23 05:55:34,029 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:34,030 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:34,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:34,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:34,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:34,032 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,032 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,033 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,033 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:34,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:34,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24f407da[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:34,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:34,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:34,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:34,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:34,036 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,036 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:34,041 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:34,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:34,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:34,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:34,048 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:34,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:34,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:34,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:34,049 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:34,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,052 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,056 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:34,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:34,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:34,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,057 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:34,057 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:34,157 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:34,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:34,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:34,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:34,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:34,160 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:34,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:34,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:34,161 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,161 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,162 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,162 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,163 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,163 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:34,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:34,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:34,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:34,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:34,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:34,164 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:34,164 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,164 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:34,164 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:34,164 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:34,164 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:34,165 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:34,165 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:34,165 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:34,165 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:34,179 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304234ec48449c366435f902b8165ce490896 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304234ec48449c366435f902b8165ce490896 2023-04-23 05:55:34,201 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/866a3b4a7b104e689df8d8280d4e4b92 2023-04-23 05:55:34,226 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/5ccb66a1cb8a4644bb0c303ae93e81dc 2023-04-23 05:55:34,239 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/6b484b8cb98e475faac256571281b2da 2023-04-23 05:55:34,245 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/866a3b4a7b104e689df8d8280d4e4b92 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/866a3b4a7b104e689df8d8280d4e4b92 2023-04-23 05:55:34,249 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/866a3b4a7b104e689df8d8280d4e4b92, entries=150, sequenceid=152, filesize=25.3 K 2023-04-23 05:55:34,249 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/5ccb66a1cb8a4644bb0c303ae93e81dc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/5ccb66a1cb8a4644bb0c303ae93e81dc 2023-04-23 05:55:34,256 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/5ccb66a1cb8a4644bb0c303ae93e81dc, entries=150, sequenceid=152, filesize=11.8 K 2023-04-23 05:55:34,256 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/6b484b8cb98e475faac256571281b2da as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/6b484b8cb98e475faac256571281b2da 2023-04-23 05:55:34,269 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/6b484b8cb98e475faac256571281b2da, entries=150, sequenceid=152, filesize=11.8 K 2023-04-23 05:55:34,270 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 105ms, sequenceid=152, compaction requested=true 2023-04-23 05:55:34,270 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:34,270 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,270 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:34,270 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:34,270 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,270 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:34,270 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,272 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,273 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,273 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:34,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dc5c8cf[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:34,275 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:34,275 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,276 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:34,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:34,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:34,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:34,283 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,283 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,283 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,284 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:34,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:34,286 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:34,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:34,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:34,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:34,286 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:34,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,290 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,294 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,294 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:34,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:34,294 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,296 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:34,296 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:34,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,396 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:34,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:34,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:34,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:34,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:34,401 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:34,401 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,401 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:34,401 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:34,401 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,401 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,403 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:34,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:34,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:34,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:34,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:34,405 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:34,405 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:34,405 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,405 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:34,405 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:34,405 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,405 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:34,405 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:34,405 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,405 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:34,405 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:34,407 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:34,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15f09cf[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:34,409 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:34,409 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,410 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:34,410 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:34,410 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:34,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:34,413 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:34,413 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,413 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:34,414 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:34,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:34,418 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:34,418 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:34,418 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:34,418 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:34,418 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:34,420 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,420 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,420 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,423 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,424 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:34,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:34,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:34,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,427 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,428 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:34,428 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:34,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:34,484 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:34,484 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:34,484 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:34,485 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:34,521 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423e069761aa78545ca8dc24689f796ed35 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e069761aa78545ca8dc24689f796ed35 2023-04-23 05:55:34,528 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:34,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:34,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:34,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:34,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:34,544 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:34,544 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,544 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:34,544 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:34,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,547 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,547 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,547 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,547 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:34,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:34,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:34,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:34,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:34,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:34,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:34,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229334551, max timestamp=1682229334565Num uniques 859; 2023-04-23 05:55:34,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:34,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:34,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:34,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229334551, max timestamp=1682229334565Num uniques 859; 2023-04-23 05:55:34,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:34,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229334551, max timestamp=1682229334565Num uniques 859; 2023-04-23 05:55:34,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:34,570 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,570 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:34,570 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=163, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/194485a0d1754bf0a4caeaf6d87bc2dc 2023-04-23 05:55:34,570 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:34,570 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,570 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:34,571 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:34,571 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,571 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:34,571 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:34,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,574 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:34,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55cc89b9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:34,577 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:34,577 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,579 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,579 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:34,579 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:34,579 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:34,579 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:34,580 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:34,580 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,580 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:34,580 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:34,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:34,584 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:34,584 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:34,584 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:34,584 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:34,584 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:34,589 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,589 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,589 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,590 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,592 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,598 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,598 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,598 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,598 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,599 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,599 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,599 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,599 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,599 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,600 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:34,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:34,600 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:34,600 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:34,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:34,619 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/ec6ef455d7954a129e19f832788a59c7 2023-04-23 05:55:34,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229394614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229394616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229394618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229394630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,700 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:34,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:34,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:34,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:34,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:34,704 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:34,704 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,704 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:34,704 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:34,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,709 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,709 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,710 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:34,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:34,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:34,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:34,714 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:34,714 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,714 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:34,716 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,716 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:34,716 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:34,716 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:34,716 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:34,718 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,718 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,719 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,719 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:34,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51029e0d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:34,722 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:34,722 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,724 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:34,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:34,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:34,725 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,725 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:34,727 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,728 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:34,728 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,728 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:34,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:34,732 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:34,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:34,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:34,732 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:34,732 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:34,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229394730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229394731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229394731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,742 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:34,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:34,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,745 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:34,746 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:34,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229394741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,846 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:34,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:34,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:34,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:34,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:34,850 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:34,850 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,850 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:34,850 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:34,850 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,850 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,851 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,851 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,852 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,852 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:34,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:34,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:34,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:34,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:34,854 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,854 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:34,854 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:34,854 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:34,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,858 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:34,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4212a752[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:34,861 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:34,861 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:34,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:34,862 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:34,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:34,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:34,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:34,869 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:34,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:34,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:34,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:34,870 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:34,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,872 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,881 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:34,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:34,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:34,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:34,881 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:34,881 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:34,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:34,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229394941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229394941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229394942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:34,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229394957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,982 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:34,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:34,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:34,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:34,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:34,985 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:34,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:34,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:34,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:34,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:34,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:34,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:34,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:34,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:34,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:34,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:34,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:34,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:34,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:34,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,991 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bfadfae[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:34,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:34,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,994 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:34,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:34,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:34,994 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:34,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:34,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:34,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:34,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:34,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:34,998 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:34,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:34,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:34,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:34,999 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:35,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:35,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:35,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:35,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:35,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:35,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:35,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:35,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:35,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,008 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:35,008 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:35,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:35,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:35,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:35,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:35,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:35,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:35,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:35,060 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/4e5e63e9424e491a806f86253728246c 2023-04-23 05:55:35,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/194485a0d1754bf0a4caeaf6d87bc2dc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/194485a0d1754bf0a4caeaf6d87bc2dc 2023-04-23 05:55:35,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/194485a0d1754bf0a4caeaf6d87bc2dc, entries=250, sequenceid=163, filesize=38.9 K 2023-04-23 05:55:35,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/ec6ef455d7954a129e19f832788a59c7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ec6ef455d7954a129e19f832788a59c7 2023-04-23 05:55:35,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ec6ef455d7954a129e19f832788a59c7, entries=150, sequenceid=163, filesize=11.8 K 2023-04-23 05:55:35,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/4e5e63e9424e491a806f86253728246c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4e5e63e9424e491a806f86253728246c 2023-04-23 05:55:35,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4e5e63e9424e491a806f86253728246c, entries=150, sequenceid=163, filesize=11.8 K 2023-04-23 05:55:35,079 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 595ms, sequenceid=163, compaction requested=true 2023-04-23 05:55:35,079 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:35,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:35,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:35,079 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:35,079 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:35,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:35,081 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117624 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:35,081 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:35,081 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/A is initiating minor compaction (all files) 2023-04-23 05:55:35,081 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/B is initiating minor compaction (all files) 2023-04-23 05:55:35,081 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/A in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:35,081 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/B in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:35,081 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/610e0f9636ce4568a5d1ca5ccf1eaffe, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3e25b8903be5475eb9ef7feff43cb9e8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/866a3b4a7b104e689df8d8280d4e4b92, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/194485a0d1754bf0a4caeaf6d87bc2dc] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=114.9 K 2023-04-23 05:55:35,081 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/171576687bb5460b88479d976c078435, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/0a1f7a2e98f642c0bac8bc18e23f54a9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/5ccb66a1cb8a4644bb0c303ae93e81dc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ec6ef455d7954a129e19f832788a59c7] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=47.1 K 2023-04-23 05:55:35,081 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 610e0f9636ce4568a5d1ca5ccf1eaffe, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682229331657 2023-04-23 05:55:35,081 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 171576687bb5460b88479d976c078435, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682229331657 2023-04-23 05:55:35,082 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 3e25b8903be5475eb9ef7feff43cb9e8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682229332053 2023-04-23 05:55:35,082 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 0a1f7a2e98f642c0bac8bc18e23f54a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682229332053 2023-04-23 05:55:35,082 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 866a3b4a7b104e689df8d8280d4e4b92, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1682229333207 2023-04-23 05:55:35,082 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 5ccb66a1cb8a4644bb0c303ae93e81dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1682229333207 2023-04-23 05:55:35,082 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 194485a0d1754bf0a4caeaf6d87bc2dc, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682229334422 2023-04-23 05:55:35,082 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ec6ef455d7954a129e19f832788a59c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682229334424 2023-04-23 05:55:35,091 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#B#compaction#326 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:35,092 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#A#compaction#325 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:35,108 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:35,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:35,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:35,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:35,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:35,111 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:35,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:35,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:35,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:35,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:35,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:35,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:35,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:35,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:35,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:35,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:35,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:35,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:35,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:35,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:35,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:35,116 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:35,116 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:35,117 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-23 05:55:35,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:35,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:35,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:35,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:35,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:35,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:35,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:35,252 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:35,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229395260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229395264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229395264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229395264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229395365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229395369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229395376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,509 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/214c1c85bad4475781962be19f78b8c9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/214c1c85bad4475781962be19f78b8c9 2023-04-23 05:55:35,515 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/B of 9b5a61538d9e2f8b71e356a41d6c31f2 into 214c1c85bad4475781962be19f78b8c9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:35,515 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:35,515 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/B, priority=12, startTime=1682229335079; duration=0sec 2023-04-23 05:55:35,515 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:35,515 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:35,517 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:35,517 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/C is initiating minor compaction (all files) 2023-04-23 05:55:35,517 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/C in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:35,517 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7d278b255af04de5990e72a626d4c4dc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/652182f73a9e457d9df6371dbffd9817, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/6b484b8cb98e475faac256571281b2da, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4e5e63e9424e491a806f86253728246c] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=47.1 K 2023-04-23 05:55:35,518 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7d278b255af04de5990e72a626d4c4dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1682229331657 2023-04-23 05:55:35,520 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 652182f73a9e457d9df6371dbffd9817, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1682229332053 2023-04-23 05:55:35,520 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 6b484b8cb98e475faac256571281b2da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1682229333207 2023-04-23 05:55:35,521 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 4e5e63e9424e491a806f86253728246c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682229334424 2023-04-23 05:55:35,535 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#C#compaction#328 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:35,546 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304237c86c5c57ad5408586fe23b14b78d6e5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304237c86c5c57ad5408586fe23b14b78d6e5 2023-04-23 05:55:35,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229395575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229395575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,583 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=200, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/5feee2ed9a4a4efa995dd718d232fd03 2023-04-23 05:55:35,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229395583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,597 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/388003a0a0264281961981bb80129dcd 2023-04-23 05:55:35,611 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=200 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/4a55b16e282f4442840ad01d0fd03e55 2023-04-23 05:55:35,616 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/5feee2ed9a4a4efa995dd718d232fd03 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/5feee2ed9a4a4efa995dd718d232fd03 2023-04-23 05:55:35,620 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/5feee2ed9a4a4efa995dd718d232fd03, entries=150, sequenceid=200, filesize=25.3 K 2023-04-23 05:55:35,621 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/388003a0a0264281961981bb80129dcd as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/388003a0a0264281961981bb80129dcd 2023-04-23 05:55:35,624 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/388003a0a0264281961981bb80129dcd, entries=150, sequenceid=200, filesize=11.8 K 2023-04-23 05:55:35,625 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/4a55b16e282f4442840ad01d0fd03e55 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4a55b16e282f4442840ad01d0fd03e55 2023-04-23 05:55:35,628 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4a55b16e282f4442840ad01d0fd03e55, entries=150, sequenceid=200, filesize=11.8 K 2023-04-23 05:55:35,629 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 512ms, sequenceid=200, compaction requested=false 2023-04-23 05:55:35,629 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:35,629 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:35,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:35,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:35,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:35,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:35,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,631 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:35,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:35,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:35,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a4a1829[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:35,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:35,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,634 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:35,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:35,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:35,635 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,635 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:35,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:35,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:35,635 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:35,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:35,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:35,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:35,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:35,640 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:35,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:35,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:35,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:35,640 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:35,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:35,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:35,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:35,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,647 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,647 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,647 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:35,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:35,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:35,647 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:35,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:35,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:35,647 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,648 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:35,648 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,648 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:35,648 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:35,648 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:35,648 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,748 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:35,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:35,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:35,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:35,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:35,752 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:35,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:35,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:35,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:35,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,753 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,753 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:35,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:35,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:35,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:35,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:35,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:35,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:35,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:35,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:35,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:35,755 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:35,756 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:35,756 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:35,756 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:35,756 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:35,756 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:35,757 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:35,757 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:35,757 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:35,757 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:35,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:35,774 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:35,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229335865, max timestamp=1682229335885Num uniques 859; 2023-04-23 05:55:35,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229335865, max timestamp=1682229335885Num uniques 859; 2023-04-23 05:55:35,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229335865, max timestamp=1682229335885Num uniques 859; 2023-04-23 05:55:35,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:35,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:35,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:35,918 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/b0dada35b8144f3198f971294bbe5ce9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/b0dada35b8144f3198f971294bbe5ce9 2023-04-23 05:55:35,924 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/A of 9b5a61538d9e2f8b71e356a41d6c31f2 into b0dada35b8144f3198f971294bbe5ce9(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:35,924 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:35,924 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/A, priority=12, startTime=1682229335079; duration=0sec 2023-04-23 05:55:35,924 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:35,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229395920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229395924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229395919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:35,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229395925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:35,966 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/b3af449c0c8a4ce4b5fb166d8603422e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b3af449c0c8a4ce4b5fb166d8603422e 2023-04-23 05:55:35,971 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/C of 9b5a61538d9e2f8b71e356a41d6c31f2 into b3af449c0c8a4ce4b5fb166d8603422e(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:35,971 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:35,972 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/C, priority=12, startTime=1682229335079; duration=0sec 2023-04-23 05:55:35,972 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:36,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229396035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229396035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229396035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57292 deadline: 1682229396036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,174 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423a0f1829933c941cea2d003cda3375ad1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a0f1829933c941cea2d003cda3375ad1 2023-04-23 05:55:36,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229396242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229396242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229396242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229396243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57282 deadline: 1682229396544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229396545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57272 deadline: 1682229396549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57298 deadline: 1682229396551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:36,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57266 deadline: 1682229396558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:36,564 DEBUG [Thread-2250] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8212 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:55:36,585 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=211, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/6a945eba2125408eb306320e18f0c47f 2023-04-23 05:55:36,597 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/fd57802a86274068a5fcb78731c42189 2023-04-23 05:55:37,038 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=211 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/442b1b8452c14a6297adf8d4b3b5b49b 2023-04-23 05:55:37,043 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/6a945eba2125408eb306320e18f0c47f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/6a945eba2125408eb306320e18f0c47f 2023-04-23 05:55:37,047 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/6a945eba2125408eb306320e18f0c47f, entries=150, sequenceid=211, filesize=25.3 K 2023-04-23 05:55:37,048 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/fd57802a86274068a5fcb78731c42189 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/fd57802a86274068a5fcb78731c42189 2023-04-23 05:55:37,051 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/fd57802a86274068a5fcb78731c42189, entries=150, sequenceid=211, filesize=11.8 K 2023-04-23 05:55:37,052 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/442b1b8452c14a6297adf8d4b3b5b49b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/442b1b8452c14a6297adf8d4b3b5b49b 2023-04-23 05:55:37,056 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/442b1b8452c14a6297adf8d4b3b5b49b, entries=150, sequenceid=211, filesize=11.8 K 2023-04-23 05:55:37,057 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 1301ms, sequenceid=211, compaction requested=true 2023-04-23 05:55:37,057 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:37,057 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:37,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:37,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:37,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:37,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:37,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-23 05:55:37,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229335918, max timestamp=1682229337057Num uniques 1018; 2023-04-23 05:55:37,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:55:37,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229335918, max timestamp=1682229337057Num uniques 1018; 2023-04-23 05:55:37,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:55:37,059 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229335918, max timestamp=1682229337057Num uniques 1018; 2023-04-23 05:55:37,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:55:37,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:37,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:37,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:37,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:37,061 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,061 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:37,061 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:37,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ffd1d9b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:37,062 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:37,062 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,072 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:37,072 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,072 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:37,074 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,074 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:37,074 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,074 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,078 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423c0a4783c21a94b0c9e3dcae3904a003e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c0a4783c21a94b0c9e3dcae3904a003e 2023-04-23 05:55:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:37,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:37,083 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:37,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:37,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:37,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:37,083 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:37,085 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,085 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,085 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,086 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,087 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:37,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:37,090 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,090 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:37,090 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:37,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682229337089, max timestamp=1682229337101Num uniques 859; 2023-04-23 05:55:37,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-23 05:55:37,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682229337089, max timestamp=1682229337101Num uniques 859; 2023-04-23 05:55:37,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682229337089, max timestamp=1682229337101Num uniques 859; 2023-04-23 05:55:37,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-23 05:55:37,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-23 05:55:37,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229397145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229397148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229397148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229397152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,190 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:37,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:37,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:37,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:37,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:37,213 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:37,213 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,213 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:37,213 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:37,214 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,214 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,217 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,217 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:37,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:37,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:37,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:37,221 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,221 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:37,221 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:37,222 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:37,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:37,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:37,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:37,224 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,225 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:37,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18820b01[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:37,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:37,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:37,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:37,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:37,231 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:37,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:37,233 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,233 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:37,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:37,237 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:37,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:37,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:37,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:37,237 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:37,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,245 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:37,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:37,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:37,245 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:37,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,246 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:37,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229397254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229397262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229397262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229397263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,346 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:37,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:37,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:37,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:37,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:37,350 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:37,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:37,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:37,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,354 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:37,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:37,356 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,356 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:37,357 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:37,357 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,364 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,364 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:37,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7568a010[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:37,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:37,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,367 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:37,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:37,367 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,369 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:37,368 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:37,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:37,371 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:37,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:37,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:37,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:37,371 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:37,373 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,373 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,373 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,374 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,377 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,381 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:37,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:37,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,381 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:37,381 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:37,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229397462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229397473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229397474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,481 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:37,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:37,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229397474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:37,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:37,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:37,485 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:37,485 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,485 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:37,485 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:37,485 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,485 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,487 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:37,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:37,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:37,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:37,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:37,488 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=252, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/4c2fccb95fee48b3a7306abd35919cec 2023-04-23 05:55:37,489 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:37,489 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:37,489 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,489 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:37,489 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:37,489 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:37,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:37,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:37,490 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:37,494 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:37,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fb965c4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:37,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:37,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,498 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:37,498 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:37,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:37,501 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:37,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:37,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:37,505 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:37,505 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:37,505 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:37,505 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:37,506 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:37,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,509 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,511 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,511 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,511 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,511 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,511 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,512 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,512 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:37,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,512 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:37,512 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,512 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,512 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:37,512 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:37,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/74841aacfab547d4bc81ddaadb532ff9 2023-04-23 05:55:37,612 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:37,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:37,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:37,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:37,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:37,616 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:37,616 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,616 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:37,616 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:37,616 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,617 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,618 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:37,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:37,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:37,620 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,620 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:37,620 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:37,620 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:37,622 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:37,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@80fa46d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:37,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,626 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,626 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:37,626 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:37,626 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:37,626 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,627 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:37,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:37,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:37,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:37,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:37,631 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:37,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:37,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:37,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:37,631 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:37,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,635 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,638 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:37,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:37,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,638 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:37,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:37,638 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:37,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,739 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:37,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:37,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:37,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:37,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:37,742 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:37,743 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,743 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:37,743 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:37,743 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,743 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:37,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:37,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:37,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:37,747 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:37,747 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,747 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:37,747 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:37,747 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:37,747 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,747 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:37,747 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:37,747 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:37,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:37,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:37,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:37,751 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:37,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd7c6e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:37,753 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:37,753 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,755 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,755 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:37,755 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:37,755 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:37,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:37,757 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:37,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:37,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:37,759 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:37,759 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:37,759 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:37,759 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:37,759 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:37,761 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,761 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,761 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,761 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:37,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:37,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,765 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,765 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:37,765 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:37,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229397769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229397779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229397780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:37,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229397784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,865 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:37,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:37,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:37,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:37,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:37,869 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:37,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:37,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:37,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,871 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,872 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,872 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:37,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:37,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:37,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:37,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:37,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:37,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:37,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:37,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:37,873 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:37,873 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,873 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:37,874 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:37,874 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:37,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:37,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:37,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:37,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,875 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:37,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:37,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30053445[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:37,880 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:37,880 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:37,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:37,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:37,882 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,882 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:37,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:37,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,884 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:37,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:37,888 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:37,888 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:37,888 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:37,888 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:37,888 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:37,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:37,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:37,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:37,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:37,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:37,896 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:37,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:37,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:37,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:37,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:37,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:37,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:37,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:37,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:37,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:37,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:37,896 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:37,897 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:37,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:37,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:37,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/25709f65c44740c0bf7b92d32cfeea79 2023-04-23 05:55:37,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/4c2fccb95fee48b3a7306abd35919cec as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4c2fccb95fee48b3a7306abd35919cec 2023-04-23 05:55:37,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4c2fccb95fee48b3a7306abd35919cec, entries=250, sequenceid=252, filesize=38.9 K 2023-04-23 05:55:37,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/74841aacfab547d4bc81ddaadb532ff9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/74841aacfab547d4bc81ddaadb532ff9 2023-04-23 05:55:37,950 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/74841aacfab547d4bc81ddaadb532ff9, entries=150, sequenceid=252, filesize=11.8 K 2023-04-23 05:55:37,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/25709f65c44740c0bf7b92d32cfeea79 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/25709f65c44740c0bf7b92d32cfeea79 2023-04-23 05:55:37,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/25709f65c44740c0bf7b92d32cfeea79, entries=150, sequenceid=252, filesize=11.8 K 2023-04-23 05:55:37,955 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=154.31 KB/158010 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 897ms, sequenceid=252, compaction requested=true 2023-04-23 05:55:37,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:37,955 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:37,955 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:37,955 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:37,955 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:37,955 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:37,957 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117910 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:37,957 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/A is initiating minor compaction (all files) 2023-04-23 05:55:37,957 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/A in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,957 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:37,957 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/b0dada35b8144f3198f971294bbe5ce9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/5feee2ed9a4a4efa995dd718d232fd03, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/6a945eba2125408eb306320e18f0c47f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4c2fccb95fee48b3a7306abd35919cec] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=115.1 K 2023-04-23 05:55:37,957 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/B is initiating minor compaction (all files) 2023-04-23 05:55:37,957 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/B in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:37,957 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/214c1c85bad4475781962be19f78b8c9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/388003a0a0264281961981bb80129dcd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/fd57802a86274068a5fcb78731c42189, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/74841aacfab547d4bc81ddaadb532ff9] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=47.3 K 2023-04-23 05:55:37,958 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b0dada35b8144f3198f971294bbe5ce9, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682229334424 2023-04-23 05:55:37,958 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 214c1c85bad4475781962be19f78b8c9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682229334424 2023-04-23 05:55:37,959 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 5feee2ed9a4a4efa995dd718d232fd03, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682229334613 2023-04-23 05:55:37,959 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 388003a0a0264281961981bb80129dcd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682229334613 2023-04-23 05:55:37,959 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 6a945eba2125408eb306320e18f0c47f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1682229335259 2023-04-23 05:55:37,959 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting fd57802a86274068a5fcb78731c42189, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1682229335259 2023-04-23 05:55:37,959 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 4c2fccb95fee48b3a7306abd35919cec, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682229335918 2023-04-23 05:55:37,960 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 74841aacfab547d4bc81ddaadb532ff9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682229335920 2023-04-23 05:55:37,972 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#B#compaction#338 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:37,974 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#A#compaction#337 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:37,994 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/092520495efe49bcba02acf6df4c0bda as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/092520495efe49bcba02acf6df4c0bda 2023-04-23 05:55:37,997 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:37,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:37,998 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/B of 9b5a61538d9e2f8b71e356a41d6c31f2 into 092520495efe49bcba02acf6df4c0bda(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:37,998 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:37,998 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/B, priority=12, startTime=1682229337955; duration=0sec 2023-04-23 05:55:37,999 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:37,999 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:38,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:38,000 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:38,000 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/C is initiating minor compaction (all files) 2023-04-23 05:55:38,000 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/C in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:38,000 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b3af449c0c8a4ce4b5fb166d8603422e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4a55b16e282f4442840ad01d0fd03e55, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/442b1b8452c14a6297adf8d4b3b5b49b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/25709f65c44740c0bf7b92d32cfeea79] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=47.3 K 2023-04-23 05:55:38,001 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b3af449c0c8a4ce4b5fb166d8603422e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1682229334424 2023-04-23 05:55:38,001 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 4a55b16e282f4442840ad01d0fd03e55, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=200, earliestPutTs=1682229334613 2023-04-23 05:55:38,001 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 442b1b8452c14a6297adf8d4b3b5b49b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=211, earliestPutTs=1682229335259 2023-04-23 05:55:38,002 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 25709f65c44740c0bf7b92d32cfeea79, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682229335920 2023-04-23 05:55:38,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:38,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:38,003 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:38,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:38,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:38,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:38,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,005 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:38,005 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:38,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:38,005 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,005 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:38,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:38,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:38,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:38,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:38,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:38,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:38,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:38,013 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#C#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:38,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:38,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:38,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:38,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:38,014 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-04-23 05:55:38,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:38,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:38,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:38,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:38,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:38,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:38,031 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/be37cc80b4ee478e9e3b483c7e987171 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/be37cc80b4ee478e9e3b483c7e987171 2023-04-23 05:55:38,035 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423cdcc7aa1a7b04b3cbfa6b15a35c67ca8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423cdcc7aa1a7b04b3cbfa6b15a35c67ca8 2023-04-23 05:55:38,035 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/C of 9b5a61538d9e2f8b71e356a41d6c31f2 into be37cc80b4ee478e9e3b483c7e987171(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:38,035 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:38,036 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/C, priority=12, startTime=1682229337955; duration=0sec 2023-04-23 05:55:38,036 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:38,040 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/3fbae1027b54441290083a93fd3e20ec 2023-04-23 05:55:38,049 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/b5bfe1e46a574c4db61ddd67f273b943 2023-04-23 05:55:38,060 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/9a7110a59d34488d8d7acf91045a4798 2023-04-23 05:55:38,064 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/3fbae1027b54441290083a93fd3e20ec as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3fbae1027b54441290083a93fd3e20ec 2023-04-23 05:55:38,067 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3fbae1027b54441290083a93fd3e20ec, entries=150, sequenceid=290, filesize=25.4 K 2023-04-23 05:55:38,068 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/b5bfe1e46a574c4db61ddd67f273b943 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b5bfe1e46a574c4db61ddd67f273b943 2023-04-23 05:55:38,073 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b5bfe1e46a574c4db61ddd67f273b943, entries=150, sequenceid=290, filesize=11.9 K 2023-04-23 05:55:38,074 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/9a7110a59d34488d8d7acf91045a4798 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9a7110a59d34488d8d7acf91045a4798 2023-04-23 05:55:38,087 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9a7110a59d34488d8d7acf91045a4798, entries=150, sequenceid=290, filesize=11.9 K 2023-04-23 05:55:38,088 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=0 B/0 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 74ms, sequenceid=290, compaction requested=false 2023-04-23 05:55:38,088 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:38,088 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:38,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:38,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:38,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:38,088 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:38,092 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,092 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:38,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:38,093 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,093 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:38,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:38,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54f6f0c4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:38,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:38,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,096 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:38,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:38,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:38,097 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,097 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:38,098 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:38,098 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:38,098 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,098 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:38,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:38,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:38,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:38,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:38,101 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:38,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:38,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:38,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:38,101 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:38,103 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:38,103 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,103 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:38,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:38,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:38,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:38,104 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:38,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,106 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,111 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:38,112 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:38,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:38,112 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:38,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:38,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:38,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:38,112 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:38,112 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:38,112 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,112 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:38,213 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:38,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:38,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:38,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:38,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:38,216 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:38,216 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:38,216 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:38,216 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:38,216 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,216 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,218 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:38,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:38,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:38,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:38,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:38,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:38,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:38,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:38,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:38,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:38,220 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:38,220 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:38,220 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:38,220 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:38,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:38,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:38,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,221 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:38,221 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:38,222 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:38,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:38,224 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:38,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21af0afb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:38,224 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,225 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,225 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:38,226 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:38,226 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:38,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:38,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:38,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:38,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:38,232 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:38,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:38,237 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:38,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:38,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:38,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:38,237 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:38,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:38,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:38,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:38,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:38,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:38,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:38,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:38,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:38,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:38,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:38,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:38,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:38,243 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:38,243 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:38,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,353 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:38,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:38,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:38,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:38,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:38,357 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:38,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:38,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:38,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:38,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,358 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:38,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:38,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:38,359 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,359 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:38,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:38,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:38,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:38,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:38,368 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:38,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:38,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:38,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:38,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:38,372 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-23 05:55:38,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:38,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:38,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:38,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:38,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:38,373 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:38,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:38,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:38,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229338430, max timestamp=1682229338453Num uniques 859; 2023-04-23 05:55:38,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229338430, max timestamp=1682229338453Num uniques 859; 2023-04-23 05:55:38,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:55:38,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229338430, max timestamp=1682229338453Num uniques 859; 2023-04-23 05:55:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:55:38,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:55:38,460 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423f92c53ee619042f5adbcfdd3d70fe286 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f92c53ee619042f5adbcfdd3d70fe286 2023-04-23 05:55:38,472 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=298, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/f4d7ce4158eb4835a08245cb373add2a 2023-04-23 05:55:38,497 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/b41415a0183c4cbb98dd90d09b84c61d 2023-04-23 05:55:38,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1682229338493, max timestamp=1682229338516Num uniques 1068; 2023-04-23 05:55:38,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1682229338493, max timestamp=1682229338516Num uniques 1068; 2023-04-23 05:55:38,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48902, new segment data size=7682, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-23 05:55:38,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48900, new segment data size=7680, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-23 05:55:38,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229338493, max timestamp=1682229338515Num uniques 1068; 2023-04-23 05:55:38,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-23 05:55:38,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229338565, max timestamp=1682229338578Num uniques 1027; 2023-04-23 05:55:38,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-23 05:55:38,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229338565, max timestamp=1682229338578Num uniques 1027; 2023-04-23 05:55:38,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-23 05:55:38,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682229338565, max timestamp=1682229338578Num uniques 1027; 2023-04-23 05:55:38,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-23 05:55:38,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682229338590, max timestamp=1682229338629Num uniques 1036; 2023-04-23 05:55:38,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-23 05:55:38,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682229338590, max timestamp=1682229338629Num uniques 1036; 2023-04-23 05:55:38,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-23 05:55:38,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682229338590, max timestamp=1682229338629Num uniques 1009; 2023-04-23 05:55:38,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-23 05:55:38,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682229338652, max timestamp=1682229338665Num uniques 1045; 2023-04-23 05:55:38,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682229338652, max timestamp=1682229338665Num uniques 1045; 2023-04-23 05:55:38,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-23 05:55:38,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-23 05:55:38,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682229338652, max timestamp=1682229338665Num uniques 1045; 2023-04-23 05:55:38,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-23 05:55:38,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682229338689, max timestamp=1682229338705Num uniques 1104; 2023-04-23 05:55:38,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-23 05:55:38,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682229338689, max timestamp=1682229338705Num uniques 1104; 2023-04-23 05:55:38,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682229338689, max timestamp=1682229338705Num uniques 1104; 2023-04-23 05:55:38,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-23 05:55:38,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-23 05:55:38,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682229338737, max timestamp=1682229338746Num uniques 1013; 2023-04-23 05:55:38,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682229338737, max timestamp=1682229338746Num uniques 1013; 2023-04-23 05:55:38,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682229338737, max timestamp=1682229338746Num uniques 1013; 2023-04-23 05:55:38,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-23 05:55:38,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-23 05:55:38,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-23 05:55:38,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1682229338770, max timestamp=1682229338793Num uniques 1072; 2023-04-23 05:55:38,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-04-23 05:55:38,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1682229338770, max timestamp=1682229338793Num uniques 1072; 2023-04-23 05:55:38,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=372, cellSize=17032, totalHeapSize=38928, min timestamp=1682229338770, max timestamp=1682229338793Num uniques 1072; 2023-04-23 05:55:38,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-04-23 05:55:38,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=17032, suffix heap size=139456,new segment heap size=38928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=372 2023-04-23 05:55:38,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8276, totalHeapSize=19064, min timestamp=1682229338819, max timestamp=1682229338838Num uniques 1231; 2023-04-23 05:55:38,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=56366, new segment data size=8276, suffix heap size=155992,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=181 2023-04-23 05:55:38,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8278, totalHeapSize=19064, min timestamp=1682229338819, max timestamp=1682229338838Num uniques 1231; 2023-04-23 05:55:38,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=56368, new segment data size=8278, suffix heap size=155992,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=181 2023-04-23 05:55:38,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8276, totalHeapSize=19064, min timestamp=1682229338819, max timestamp=1682229338838Num uniques 1231; 2023-04-23 05:55:38,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=56366, new segment data size=8276, suffix heap size=155992,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1231, new segment cells count=181 2023-04-23 05:55:38,861 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/4d90cf47fa6044e1a2a18aaeba7e8bd7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4d90cf47fa6044e1a2a18aaeba7e8bd7 2023-04-23 05:55:38,868 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/A of 9b5a61538d9e2f8b71e356a41d6c31f2 into 4d90cf47fa6044e1a2a18aaeba7e8bd7(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:38,868 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:38,868 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/A, priority=12, startTime=1682229337955; duration=0sec 2023-04-23 05:55:38,868 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:38,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682229338861, max timestamp=1682229338909Num uniques 1040; 2023-04-23 05:55:38,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682229338861, max timestamp=1682229338909Num uniques 1040; 2023-04-23 05:55:38,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-23 05:55:38,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1682229338861, max timestamp=1682229338909Num uniques 1040; 2023-04-23 05:55:38,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-23 05:55:38,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-23 05:55:38,925 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/c3ef007ed6d146338da8e83566c9969f 2023-04-23 05:55:38,930 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/f4d7ce4158eb4835a08245cb373add2a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f4d7ce4158eb4835a08245cb373add2a 2023-04-23 05:55:38,934 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f4d7ce4158eb4835a08245cb373add2a, entries=150, sequenceid=298, filesize=25.4 K 2023-04-23 05:55:38,935 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/b41415a0183c4cbb98dd90d09b84c61d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b41415a0183c4cbb98dd90d09b84c61d 2023-04-23 05:55:38,938 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b41415a0183c4cbb98dd90d09b84c61d, entries=100, sequenceid=298, filesize=9.5 K 2023-04-23 05:55:38,939 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/c3ef007ed6d146338da8e83566c9969f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c3ef007ed6d146338da8e83566c9969f 2023-04-23 05:55:38,942 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c3ef007ed6d146338da8e83566c9969f, entries=100, sequenceid=298, filesize=9.5 K 2023-04-23 05:55:38,943 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=114.05 KB/116790 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 571ms, sequenceid=298, compaction requested=true 2023-04-23 05:55:38,943 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:38,943 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:38,943 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:38,943 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:38,943 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:38,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:38,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:38,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=127.47 KB heapSize=357.47 KB 2023-04-23 05:55:38,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:38,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,945 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:38,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:38,945 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:38,946 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:38,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:38,946 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:38,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:38,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:38,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7a2cac[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:38,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:38,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,952 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:38,952 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:38,956 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,956 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:38,958 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:38,958 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,958 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:38,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:38,958 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:38,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:38,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:38,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:38,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:38,965 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304233fc0d333b4d44f39853bd384f674b8de to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233fc0d333b4d44f39853bd384f674b8de 2023-04-23 05:55:38,964 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:38,965 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:38,965 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:38,965 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:38,965 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:38,969 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,969 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:38,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:38,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:38,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:38,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229398961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,973 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:38,974 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:38,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,976 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:38,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:38,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229398970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,979 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=489, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/8cd5ee94c98d4e78aa69474cd84f883b 2023-04-23 05:55:38,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:38,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229398971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:38,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:38,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229398971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:38,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:38,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:38,981 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:38,982 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:38,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:38,982 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:38,982 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:38,982 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:38,984 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,984 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:38,984 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:38,984 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,984 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:38,984 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/698fd6c1d64d407a92c60d3f76e26524 2023-04-23 05:55:39,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229399072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,082 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:39,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:39,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229399081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:39,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:39,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:39,091 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:39,091 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:39,092 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:39,092 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:39,092 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,092 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,093 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,093 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:39,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:39,094 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,094 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:39,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:39,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:39,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:39,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:39,096 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:39,096 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:39,096 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:39,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229399088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229399089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,109 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:39,109 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:39,109 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,109 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:39,109 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:39,109 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,109 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:39,109 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:39,109 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,109 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:39,109 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:39,111 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:39,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:39,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:39,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:39,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53449eba[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:39,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:39,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,116 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:39,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:39,118 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:39,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:39,121 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:39,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:39,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:39,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:39,121 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:39,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:39,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,123 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:39,125 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,129 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:39,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:39,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:39,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,130 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:39,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:39,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:39,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:39,131 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:39,131 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:39,231 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:39,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:39,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:39,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:39,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:39,234 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:39,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:39,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:39,235 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:39,235 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,235 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,237 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:39,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:39,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:39,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:39,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:39,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:39,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:39,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:39,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:39,243 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,243 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:39,243 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:39,243 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:39,245 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:39,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:39,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@305724c6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:39,247 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:39,247 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:39,249 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,249 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:39,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,250 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:39,251 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:39,251 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:39,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:39,255 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:39,255 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:39,255 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:39,255 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:39,255 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:39,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:39,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,268 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:39,269 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:39,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,270 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:39,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,276 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:39,276 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:39,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:39,276 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:39,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:39,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:39,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:39,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:39,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:39,276 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:39,276 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:39,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229399275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229399287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229399299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229399299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,376 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:39,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:39,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:39,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:39,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:39,380 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:39,380 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:39,380 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:39,380 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:39,380 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,380 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,381 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:39,381 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:39,382 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,382 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:39,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:39,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:39,383 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,383 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:39,383 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:39,383 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:39,385 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,385 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:39,385 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,385 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@697618e5[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:39,386 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:39,387 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,387 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,388 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:39,388 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:39,388 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:39,388 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,388 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:39,390 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:39,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:39,390 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:39,392 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:39,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:39,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:39,393 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:39,393 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:39,394 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,394 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:39,394 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:39,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:39,394 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:39,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:39,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:39,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:39,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:39,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:39,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:39,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:39,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:39,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:39,398 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:39,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:39,398 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,398 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:39,398 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,398 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:39,398 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:39,398 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/974f0ab7448f4df284cbf96eda7b7cde 2023-04-23 05:55:39,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/8cd5ee94c98d4e78aa69474cd84f883b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/8cd5ee94c98d4e78aa69474cd84f883b 2023-04-23 05:55:39,428 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/8cd5ee94c98d4e78aa69474cd84f883b, entries=300, sequenceid=489, filesize=46.1 K 2023-04-23 05:55:39,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/698fd6c1d64d407a92c60d3f76e26524 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/698fd6c1d64d407a92c60d3f76e26524 2023-04-23 05:55:39,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/698fd6c1d64d407a92c60d3f76e26524, entries=150, sequenceid=489, filesize=11.9 K 2023-04-23 05:55:39,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/974f0ab7448f4df284cbf96eda7b7cde as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/974f0ab7448f4df284cbf96eda7b7cde 2023-04-23 05:55:39,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/974f0ab7448f4df284cbf96eda7b7cde, entries=150, sequenceid=489, filesize=11.9 K 2023-04-23 05:55:39,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~377.34 KB/386400, currentSize=53.67 KB/54960 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 493ms, sequenceid=489, compaction requested=true 2023-04-23 05:55:39,437 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:39,437 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:39,437 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:39,437 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:39,437 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:39,437 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:39,439 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 125632 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:39,439 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46628 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:39,439 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/A is initiating minor compaction (all files) 2023-04-23 05:55:39,439 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/B is initiating minor compaction (all files) 2023-04-23 05:55:39,439 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/A in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,439 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/B in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,439 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4d90cf47fa6044e1a2a18aaeba7e8bd7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3fbae1027b54441290083a93fd3e20ec, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f4d7ce4158eb4835a08245cb373add2a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/8cd5ee94c98d4e78aa69474cd84f883b] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=122.7 K 2023-04-23 05:55:39,439 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/092520495efe49bcba02acf6df4c0bda, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b5bfe1e46a574c4db61ddd67f273b943, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b41415a0183c4cbb98dd90d09b84c61d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/698fd6c1d64d407a92c60d3f76e26524] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=45.5 K 2023-04-23 05:55:39,439 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 4d90cf47fa6044e1a2a18aaeba7e8bd7, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682229335920 2023-04-23 05:55:39,440 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 092520495efe49bcba02acf6df4c0bda, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682229335920 2023-04-23 05:55:39,440 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 3fbae1027b54441290083a93fd3e20ec, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1682229337139 2023-04-23 05:55:39,440 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b5bfe1e46a574c4db61ddd67f273b943, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1682229337139 2023-04-23 05:55:39,440 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting f4d7ce4158eb4835a08245cb373add2a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682229338331 2023-04-23 05:55:39,440 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b41415a0183c4cbb98dd90d09b84c61d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682229338332 2023-04-23 05:55:39,440 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 8cd5ee94c98d4e78aa69474cd84f883b, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682229338928 2023-04-23 05:55:39,440 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 698fd6c1d64d407a92c60d3f76e26524, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682229338942 2023-04-23 05:55:39,461 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#B#compaction#349 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:39,493 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#A#compaction#350 average throughput is 0.60 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:39,498 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:39,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:39,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:39,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:39,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:39,516 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:39,516 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:39,516 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:39,516 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:39,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:39,518 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:39,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,519 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,519 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:39,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:39,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:39,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:39,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:39,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:39,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:39,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:39,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:39,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:39,528 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:39,528 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,528 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:39,528 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:39,528 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:55:39,528 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:39,529 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:39,529 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:39,529 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:39,529 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:39,529 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:39,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:39,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682229339608, max timestamp=1682229339623Num uniques 859; 2023-04-23 05:55:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682229339608, max timestamp=1682229339623Num uniques 859; 2023-04-23 05:55:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-23 05:55:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1682229339608, max timestamp=1682229339623Num uniques 859; 2023-04-23 05:55:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-23 05:55:39,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-23 05:55:39,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229399648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229399656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229399656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229399657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229399758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229399765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229399765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229399770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,891 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/e1bdf4434f9c489dacafd38d59a94841 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e1bdf4434f9c489dacafd38d59a94841 2023-04-23 05:55:39,896 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/B of 9b5a61538d9e2f8b71e356a41d6c31f2 into e1bdf4434f9c489dacafd38d59a94841(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:39,896 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:39,896 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/B, priority=12, startTime=1682229339437; duration=0sec 2023-04-23 05:55:39,896 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:39,896 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:39,898 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46628 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:39,898 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/C is initiating minor compaction (all files) 2023-04-23 05:55:39,898 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/C in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:39,898 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/be37cc80b4ee478e9e3b483c7e987171, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9a7110a59d34488d8d7acf91045a4798, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c3ef007ed6d146338da8e83566c9969f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/974f0ab7448f4df284cbf96eda7b7cde] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=45.5 K 2023-04-23 05:55:39,898 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting be37cc80b4ee478e9e3b483c7e987171, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1682229335920 2023-04-23 05:55:39,898 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9a7110a59d34488d8d7acf91045a4798, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1682229337139 2023-04-23 05:55:39,899 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c3ef007ed6d146338da8e83566c9969f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1682229338332 2023-04-23 05:55:39,899 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 974f0ab7448f4df284cbf96eda7b7cde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682229338942 2023-04-23 05:55:39,907 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#C#compaction#352 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:39,948 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d7ce2cf272624e0bafa4ae6c1886f69a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d7ce2cf272624e0bafa4ae6c1886f69a 2023-04-23 05:55:39,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229399965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229399975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229399976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:39,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:39,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229399976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229400275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229400286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229400288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229400288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,319 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/9c465361489d46bd9f15051f2a1ffcad as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9c465361489d46bd9f15051f2a1ffcad 2023-04-23 05:55:40,324 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/C of 9b5a61538d9e2f8b71e356a41d6c31f2 into 9c465361489d46bd9f15051f2a1ffcad(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:40,325 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:40,325 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/C, priority=12, startTime=1682229339437; duration=0sec 2023-04-23 05:55:40,325 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:40,346 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/fb08084ef70f4e0b9e35b6c385522c32 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/fb08084ef70f4e0b9e35b6c385522c32 2023-04-23 05:55:40,353 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/A of 9b5a61538d9e2f8b71e356a41d6c31f2 into fb08084ef70f4e0b9e35b6c385522c32(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:40,353 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:40,353 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/A, priority=12, startTime=1682229339437; duration=0sec 2023-04-23 05:55:40,353 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:40,355 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=500, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/e6759161155f44cf8076606b98418ee2 2023-04-23 05:55:40,399 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/7de09abd00fa43c0a82a2fa88d1c7652 2023-04-23 05:55:40,435 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/7038173c36f34afca3cac6f2ee5ffc89 2023-04-23 05:55:40,440 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/e6759161155f44cf8076606b98418ee2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/e6759161155f44cf8076606b98418ee2 2023-04-23 05:55:40,444 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/e6759161155f44cf8076606b98418ee2, entries=150, sequenceid=500, filesize=25.4 K 2023-04-23 05:55:40,445 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/7de09abd00fa43c0a82a2fa88d1c7652 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/7de09abd00fa43c0a82a2fa88d1c7652 2023-04-23 05:55:40,448 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/7de09abd00fa43c0a82a2fa88d1c7652, entries=150, sequenceid=500, filesize=11.9 K 2023-04-23 05:55:40,449 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/7038173c36f34afca3cac6f2ee5ffc89 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7038173c36f34afca3cac6f2ee5ffc89 2023-04-23 05:55:40,453 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7038173c36f34afca3cac6f2ee5ffc89, entries=150, sequenceid=500, filesize=11.9 K 2023-04-23 05:55:40,454 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 926ms, sequenceid=500, compaction requested=false 2023-04-23 05:55:40,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:40,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,455 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:40,455 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:40,457 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,457 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:40,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:40,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:40,459 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:40,459 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:40,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:40,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:40,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72f2462b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:40,460 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:40,460 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:40,461 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:40,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:40,462 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:40,462 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:40,462 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:40,462 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,462 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:40,463 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:40,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:40,463 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:40,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:40,466 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:40,466 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:40,466 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:40,466 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:40,466 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:40,467 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:40,467 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:40,467 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:40,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:40,468 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:40,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:40,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:40,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:40,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:40,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:40,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:40,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:40,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:40,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:40,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:40,472 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:40,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:40,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:40,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:40,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:40,473 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:40,473 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:40,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:40,573 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:40,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:40,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:40,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:40,576 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:40,576 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:40,576 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:40,576 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:40,577 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:40,577 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:40,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:40,578 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:40,578 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,578 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:40,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:40,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:40,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:40,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:40,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:40,579 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:40,579 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:40,579 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:40,580 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:40,580 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:40,580 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-04-23 05:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:40,580 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:40,581 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:40,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:40,788 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:40,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229400803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229400804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229400810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229400810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229400911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229400911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229400911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:40,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229400911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:40,993 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042329e1c038904a4252837dcc6f411a2554 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042329e1c038904a4252837dcc6f411a2554 2023-04-23 05:55:40,998 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=536, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/551aabe864d84d4ab48096fc066b81de 2023-04-23 05:55:41,007 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/9518eb2e7b464acc8d3b729c66e8c000 2023-04-23 05:55:41,019 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/649eddd244dc453984fa082dabafaf18 2023-04-23 05:55:41,023 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/551aabe864d84d4ab48096fc066b81de as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/551aabe864d84d4ab48096fc066b81de 2023-04-23 05:55:41,026 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/551aabe864d84d4ab48096fc066b81de, entries=150, sequenceid=536, filesize=25.4 K 2023-04-23 05:55:41,027 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/9518eb2e7b464acc8d3b729c66e8c000 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/9518eb2e7b464acc8d3b729c66e8c000 2023-04-23 05:55:41,030 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/9518eb2e7b464acc8d3b729c66e8c000, entries=150, sequenceid=536, filesize=11.9 K 2023-04-23 05:55:41,031 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/649eddd244dc453984fa082dabafaf18 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/649eddd244dc453984fa082dabafaf18 2023-04-23 05:55:41,034 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/649eddd244dc453984fa082dabafaf18, entries=150, sequenceid=536, filesize=11.9 K 2023-04-23 05:55:41,035 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=60.38 KB/61830 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 455ms, sequenceid=536, compaction requested=true 2023-04-23 05:55:41,035 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:41,035 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:41,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:41,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:41,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:41,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,037 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,038 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:41,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4a781[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:41,038 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:41,038 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,039 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:41,039 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,039 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:41,039 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:41,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:41,045 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:41,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:41,047 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:41,047 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:41,047 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:41,047 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:41,048 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:41,049 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,049 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,049 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,050 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:41,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:41,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,053 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,053 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,054 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:41,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,054 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:41,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,054 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:41,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:41,122 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-23 05:55:41,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:41,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:41,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:41,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:41,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:41,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:41,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229401146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229401147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229401147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229401147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,154 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:41,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:41,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:41,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:41,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:41,157 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:41,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:41,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:41,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,158 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:41,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:41,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,161 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:41,162 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,162 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:41,162 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:41,162 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:41,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:41,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,163 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:41,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cccc2e5[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:41,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:41,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,166 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,166 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:41,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:41,166 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:41,166 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:41,168 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:41,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:41,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:41,170 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:41,170 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:41,170 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:41,170 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:41,170 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:41,173 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,173 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,173 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,174 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,176 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,180 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:41,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:41,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,181 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:41,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,181 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:41,181 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,181 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,181 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229401248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229401254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229401254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229401254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,281 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:41,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:41,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:41,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:41,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:41,284 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:41,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:41,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:41,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,287 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:41,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:41,289 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:41,289 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,289 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:41,289 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:41,289 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:41,289 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,289 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:41,290 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:41,290 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:41,291 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,291 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:41,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:41,293 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:41,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55a442a9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:41,293 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,295 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,295 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:41,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:41,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:41,296 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,296 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:41,297 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,297 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,297 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,297 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:41,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:41,299 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:41,300 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:41,300 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:41,300 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:41,300 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:41,301 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,301 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,301 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,302 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,303 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:41,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:41,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,305 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:41,305 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:41,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,405 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:41,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:41,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:41,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:41,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:41,409 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:41,409 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,409 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:41,409 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:41,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:41,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:41,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:41,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:41,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:41,413 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:41,413 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,413 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:41,413 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:41,413 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:41,413 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,413 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:41,414 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:41,414 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:41,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:41,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:41,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:41,415 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,415 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:41,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35078528[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:41,417 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:41,417 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,418 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:41,418 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:41,418 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:41,418 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,418 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:41,419 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,419 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,419 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,420 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:41,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:41,422 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:41,422 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:41,422 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:41,422 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:41,422 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:41,424 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,424 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,424 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,425 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,426 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:41,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:41,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,428 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,429 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:41,429 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,429 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:41,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229401455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229401460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229401460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229401460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,529 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:41,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:41,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:41,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:41,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:41,532 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:41,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:41,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:41,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,536 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304234a2bdfa6127a4ffc8c9e7100ea6796ba to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304234a2bdfa6127a4ffc8c9e7100ea6796ba 2023-04-23 05:55:41,541 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,542 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:41,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:41,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,543 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,543 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:41,543 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,543 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:41,543 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:41,543 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:41,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:41,546 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,547 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,547 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,547 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:41,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152fb4a6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:41,549 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:41,549 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,550 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:41,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:41,550 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,550 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:41,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:41,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:41,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,552 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:41,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:41,556 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:41,556 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:41,556 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:41,556 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:41,556 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:41,557 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,557 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,558 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,562 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,562 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,562 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:41,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:41,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,563 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,563 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:41,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,563 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:41,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,663 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:41,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:41,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:41,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:41,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:41,667 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:41,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:41,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:41,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,668 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,668 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,668 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,668 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:41,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:41,669 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:41,670 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,670 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:41,670 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:41,670 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:41,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:41,671 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,672 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,672 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,672 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:41,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9e870c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:41,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:41,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:41,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:41,675 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,675 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:41,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:41,677 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:41,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:41,679 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:41,680 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:41,680 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:41,680 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:41,680 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:41,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,682 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:41,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:41,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:41,686 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:41,686 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:41,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229401760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229401765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229401768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:41,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229401768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,786 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:41,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:41,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:41,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:41,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:41,790 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:41,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:41,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:41,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,792 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,792 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,792 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,792 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:41,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:41,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:41,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:41,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:41,793 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:41,793 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,793 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:41,794 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:41,794 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:41,794 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,794 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:41,794 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:41,795 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:41,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,800 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,801 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,801 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:41,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:41,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d8335c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:41,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:41,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:41,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:41,808 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,808 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:41,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:41,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:41,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:41,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:41,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:41,816 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:41,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:41,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:41,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:41,816 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:41,818 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,818 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,818 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,827 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,827 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:41,827 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:41,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,827 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,828 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:41,828 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,840 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:41,840 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,940 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:41,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:41,942 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=551, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/368e090aeb134245ab0e3962cbca76b2 2023-04-23 05:55:41,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:41,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:41,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:41,948 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:41,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:41,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:41,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,950 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,950 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,950 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:41,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:41,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:41,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:41,956 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:41,956 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:41,956 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:41,964 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:41,964 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:41,965 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,965 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:41,965 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:41,965 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:41,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,969 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17fc9aa[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:41,970 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:41,971 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,976 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,977 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:41,977 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:41,977 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:41,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:41,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:41,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,978 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:41,982 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:41,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:41,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:41,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:41,982 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:41,984 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:41,985 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,985 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:41,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,988 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,988 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:41,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:41,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:41,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:41,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:41,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:41,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:41,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:41,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:41,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:41,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:41,989 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:41,989 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:41,989 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:42,090 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:42,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:42,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:42,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:42,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:42,093 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:42,093 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:42,093 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:42,093 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:42,093 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,093 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,095 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:42,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:42,096 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,096 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:42,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:42,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:42,097 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,097 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:42,097 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:42,097 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:42,097 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:42,098 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,098 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:42,098 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:42,099 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,099 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:42,099 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:42,099 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:42,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:42,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699dc7a7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:42,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:42,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,102 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:42,102 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,102 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:42,103 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:42,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:42,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:42,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:42,106 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:42,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:42,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:42,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:42,107 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:42,109 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,109 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,109 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:42,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:42,110 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:42,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:42,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:42,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:42,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:42,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:42,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:42,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:42,113 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:42,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:42,116 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:42,116 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:42,216 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:42,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:42,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:42,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:42,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:42,219 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:42,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:42,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:42,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:42,220 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,220 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,221 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,221 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:42,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:42,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:42,221 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,221 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:42,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:42,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:42,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:42,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:42,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:42,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:42,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:42,222 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:42,222 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,223 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:42,223 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:42,223 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:42,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:42,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:42,223 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:42,224 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,224 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:42,224 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,224 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:42,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:42,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:42,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b08b7f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:42,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:42,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:42,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:42,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,229 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:42,229 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:42,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:42,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:42,229 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:42,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:42,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:42,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:42,231 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:42,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:42,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:42,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:42,231 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:42,233 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,233 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:42,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,233 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:42,233 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,236 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,236 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,236 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,236 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:42,236 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:42,236 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:42,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:42,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:42,237 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,237 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,237 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:42,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:42,237 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:42,237 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:42,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:42,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:42,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:42,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:42,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229402272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:42,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229402275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:42,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229402276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:42,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229402277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,337 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:42,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:42,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:42,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:42,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:42,340 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:42,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:42,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:42,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:42,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,344 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,344 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:42,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:42,344 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,344 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:42,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:42,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:42,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:42,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:42,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:42,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:42,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:42,345 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,345 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:42,346 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:42,346 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:42,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:42,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:42,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:42,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,347 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:42,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:42,347 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,348 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:42,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bbb06a6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:42,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:42,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,350 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,350 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:42,350 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:42,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:42,350 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:42,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,355 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:42,355 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:42,355 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:42,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:42,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:42,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:42,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:42,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:42,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:42,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:42,358 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:42,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:42,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:42,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:42,358 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:42,359 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,359 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,359 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:42,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:42,360 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:42,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,361 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:42,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:42,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:42,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:42,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:42,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:42,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:42,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:42,363 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:42,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:42,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:42,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:42,364 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:42,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,364 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:42,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/bd97c410e8d64de499dd6559df0fb806 2023-04-23 05:55:42,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/e619f912a36a4fe0912db2ac9f41abef 2023-04-23 05:55:42,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/368e090aeb134245ab0e3962cbca76b2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/368e090aeb134245ab0e3962cbca76b2 2023-04-23 05:55:42,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/368e090aeb134245ab0e3962cbca76b2, entries=200, sequenceid=551, filesize=32.3 K 2023-04-23 05:55:42,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/bd97c410e8d64de499dd6559df0fb806 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/bd97c410e8d64de499dd6559df0fb806 2023-04-23 05:55:42,388 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/bd97c410e8d64de499dd6559df0fb806, entries=150, sequenceid=551, filesize=11.9 K 2023-04-23 05:55:42,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/e619f912a36a4fe0912db2ac9f41abef as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/e619f912a36a4fe0912db2ac9f41abef 2023-04-23 05:55:42,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/e619f912a36a4fe0912db2ac9f41abef, entries=150, sequenceid=551, filesize=11.9 K 2023-04-23 05:55:42,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=107.34 KB/109920 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 1270ms, sequenceid=551, compaction requested=true 2023-04-23 05:55:42,392 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:42,392 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:42,392 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:42,392 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:42,392 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:42,392 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:42,393 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111838 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:42,393 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/A is initiating minor compaction (all files) 2023-04-23 05:55:42,393 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/A in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,393 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/fb08084ef70f4e0b9e35b6c385522c32, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/e6759161155f44cf8076606b98418ee2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/551aabe864d84d4ab48096fc066b81de, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/368e090aeb134245ab0e3962cbca76b2] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=109.2 K 2023-04-23 05:55:42,394 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:42,394 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting fb08084ef70f4e0b9e35b6c385522c32, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682229338942 2023-04-23 05:55:42,394 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/B is initiating minor compaction (all files) 2023-04-23 05:55:42,394 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/B in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,394 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e1bdf4434f9c489dacafd38d59a94841, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/7de09abd00fa43c0a82a2fa88d1c7652, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/9518eb2e7b464acc8d3b729c66e8c000, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/bd97c410e8d64de499dd6559df0fb806] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=48.2 K 2023-04-23 05:55:42,394 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e6759161155f44cf8076606b98418ee2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1682229338958 2023-04-23 05:55:42,394 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e1bdf4434f9c489dacafd38d59a94841, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682229338942 2023-04-23 05:55:42,394 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 551aabe864d84d4ab48096fc066b81de, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682229339639 2023-04-23 05:55:42,395 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7de09abd00fa43c0a82a2fa88d1c7652, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1682229338958 2023-04-23 05:55:42,395 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 368e090aeb134245ab0e3962cbca76b2, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1682229340802 2023-04-23 05:55:42,395 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9518eb2e7b464acc8d3b729c66e8c000, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682229339639 2023-04-23 05:55:42,395 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting bd97c410e8d64de499dd6559df0fb806, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1682229340802 2023-04-23 05:55:42,403 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#B#compaction#362 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:42,404 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#A#compaction#361 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:42,430 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/d7842a2f4aab40f0827e14bf5eedbeca as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/d7842a2f4aab40f0827e14bf5eedbeca 2023-04-23 05:55:42,442 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/B of 9b5a61538d9e2f8b71e356a41d6c31f2 into d7842a2f4aab40f0827e14bf5eedbeca(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:42,442 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:42,442 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/B, priority=12, startTime=1682229342392; duration=0sec 2023-04-23 05:55:42,443 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:42,443 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:42,444 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:42,444 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/C is initiating minor compaction (all files) 2023-04-23 05:55:42,444 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/C in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,444 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9c465361489d46bd9f15051f2a1ffcad, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7038173c36f34afca3cac6f2ee5ffc89, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/649eddd244dc453984fa082dabafaf18, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/e619f912a36a4fe0912db2ac9f41abef] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=48.2 K 2023-04-23 05:55:42,445 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9c465361489d46bd9f15051f2a1ffcad, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1682229338942 2023-04-23 05:55:42,445 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7038173c36f34afca3cac6f2ee5ffc89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1682229338958 2023-04-23 05:55:42,445 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 649eddd244dc453984fa082dabafaf18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1682229339639 2023-04-23 05:55:42,446 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e619f912a36a4fe0912db2ac9f41abef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1682229340802 2023-04-23 05:55:42,453 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#C#compaction#363 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:42,464 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:42,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:42,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:42,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:42,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:42,469 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:42,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:42,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:42,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:42,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,469 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,470 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:42,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:42,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:42,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:42,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:42,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:42,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:42,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:42,480 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:42,480 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:42,480 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:42,480 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:42,480 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:42,480 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,480 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:42,481 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-23 05:55:42,481 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:42,481 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:42,481 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:42,481 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:42,481 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:42,481 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:42,515 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423a9bb134954ca4fdc992c1fc9f06150fa to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a9bb134954ca4fdc992c1fc9f06150fa 2023-04-23 05:55:42,530 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=571, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/f772062291014a4499c5cc6dfd0845df 2023-04-23 05:55:42,851 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/4ab30ac1d7e840e4abea019aa50fd758 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4ab30ac1d7e840e4abea019aa50fd758 2023-04-23 05:55:42,863 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/A of 9b5a61538d9e2f8b71e356a41d6c31f2 into 4ab30ac1d7e840e4abea019aa50fd758(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:42,863 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:42,863 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/A, priority=12, startTime=1682229342392; duration=0sec 2023-04-23 05:55:42,863 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:42,875 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/b4a6105d2ce04d18b79711578d12613f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b4a6105d2ce04d18b79711578d12613f 2023-04-23 05:55:42,884 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/C of 9b5a61538d9e2f8b71e356a41d6c31f2 into b4a6105d2ce04d18b79711578d12613f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:42,884 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:42,884 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/C, priority=12, startTime=1682229342392; duration=0sec 2023-04-23 05:55:42,884 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:42,946 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/4aa76702cf314de2a3a9dda853539ed2 2023-04-23 05:55:42,969 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/0f5552dac68b4e8090aad42e6b7cb70f 2023-04-23 05:55:42,974 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/f772062291014a4499c5cc6dfd0845df as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f772062291014a4499c5cc6dfd0845df 2023-04-23 05:55:42,978 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f772062291014a4499c5cc6dfd0845df, entries=150, sequenceid=571, filesize=25.4 K 2023-04-23 05:55:42,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/4aa76702cf314de2a3a9dda853539ed2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/4aa76702cf314de2a3a9dda853539ed2 2023-04-23 05:55:42,986 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/4aa76702cf314de2a3a9dda853539ed2, entries=150, sequenceid=571, filesize=11.9 K 2023-04-23 05:55:42,987 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/0f5552dac68b4e8090aad42e6b7cb70f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/0f5552dac68b4e8090aad42e6b7cb70f 2023-04-23 05:55:42,997 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/0f5552dac68b4e8090aad42e6b7cb70f, entries=150, sequenceid=571, filesize=11.9 K 2023-04-23 05:55:42,998 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 518ms, sequenceid=571, compaction requested=false 2023-04-23 05:55:42,998 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:42,998 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:42,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:42,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:42,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:42,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:42,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:43,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:43,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5476b8e2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:43,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:43,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:43,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:43,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:43,010 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,010 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:43,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:43,022 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:43,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:43,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:43,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:43,022 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:43,023 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,023 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,023 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,025 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,026 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,031 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:43,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:43,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,031 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:43,031 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:43,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,032 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,032 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,032 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,032 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,131 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:43,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:43,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:43,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:43,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:43,140 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:43,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:43,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:43,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,141 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,142 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:43,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:43,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:43,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:43,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:43,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:43,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,143 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:43,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:43,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:43,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:43,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:43,144 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:43,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:43,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:43,145 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:43,146 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:43,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5351ed34[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:43,148 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:43,148 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,149 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:43,150 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:43,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:43,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,157 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:43,157 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,157 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:43,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:43,160 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:43,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:43,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:43,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:43,160 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:43,162 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,162 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,162 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,162 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,164 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,166 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,166 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,166 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:43,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:43,166 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:43,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,167 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:43,167 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:43,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,267 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:43,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:43,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:43,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:43,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:43,271 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:43,271 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,271 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:43,271 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:43,271 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,272 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,273 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,273 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,273 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,273 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:43,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:43,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:43,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:43,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:43,276 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,276 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:43,276 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:43,277 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:43,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:43,279 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,279 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,279 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,279 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:43,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:43,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35dc6f52[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:43,281 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:43,281 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,282 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:43,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:43,282 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:43,283 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:43,284 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,284 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:43,284 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,284 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:43,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:43,287 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:43,287 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:43,287 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:43,287 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:43,287 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:43,292 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,292 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,292 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,293 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,294 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:43,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,297 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:43,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:43,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,297 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,297 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,297 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:43,297 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:43,297 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,297 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:43,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:43,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:43,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:43,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:43,378 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:43,378 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:43,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:43,393 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423e6e3af0381af4a86acd14fc6e45c2a53 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e6e3af0381af4a86acd14fc6e45c2a53 2023-04-23 05:55:43,397 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:43,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:43,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:43,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:43,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:43,402 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:43,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:43,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:43,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,404 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:43,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:43,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:43,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:43,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:43,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:43,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:43,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:43,408 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,408 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:43,408 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:43,408 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,409 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:43,409 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:43,409 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,409 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:43,409 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:43,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,410 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,411 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,411 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:43,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:43,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4621c179[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:43,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,415 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:43,416 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,416 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:43,417 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,417 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:43,419 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:43,419 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:43,419 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:43,419 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:43,419 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:43,421 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,421 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,421 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,421 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,423 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,425 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,426 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,426 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,426 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,426 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,426 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,426 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:43,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:43,426 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,426 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:43,426 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:43,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:43,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229343413, max timestamp=1682229343424Num uniques 859; 2023-04-23 05:55:43,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:43,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229343413, max timestamp=1682229343424Num uniques 859; 2023-04-23 05:55:43,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:43,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,429 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229343413, max timestamp=1682229343424Num uniques 859; 2023-04-23 05:55:43,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:43,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682229343450, max timestamp=1682229343467Num uniques 1018; 2023-04-23 05:55:43,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682229343450, max timestamp=1682229343467Num uniques 1018; 2023-04-23 05:55:43,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-23 05:55:43,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-23 05:55:43,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229343450, max timestamp=1682229343467Num uniques 1018; 2023-04-23 05:55:43,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:55:43,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229403494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229403505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229403506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229403507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,526 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:43,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:43,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:43,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:43,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:43,530 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:43,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:43,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:43,531 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,531 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,538 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,538 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,538 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,538 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:43,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:43,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:43,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:43,539 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:43,539 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,539 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:43,539 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:43,539 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:43,540 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,540 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:43,540 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:43,540 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:43,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:43,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:43,540 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:43,541 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,542 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,542 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:43,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:43,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2771c0f7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:43,543 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:43,543 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,544 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,544 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:43,544 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:43,544 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:43,544 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:43,546 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,546 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:43,546 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,546 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:43,548 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:43,548 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:43,548 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:43,548 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:43,548 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:43,550 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,550 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,550 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,552 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,555 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,555 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,555 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,555 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,555 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,555 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,555 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,555 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,555 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:43,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:43,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:43,555 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,556 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:43,556 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:43,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229403608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229403618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229403621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229403623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,656 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:43,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:43,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:43,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:43,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:43,664 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:43,664 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,664 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:43,664 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:43,665 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,665 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,666 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,667 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,667 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:43,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:43,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:43,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:43,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:43,672 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:43,672 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,672 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:43,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:43,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:43,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:43,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:43,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:43,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:43,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:43,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:43,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:43,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:43,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b15b160[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:43,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:43,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,688 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,688 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:43,688 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:43,688 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:43,689 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,689 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,691 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:43,691 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:43,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:43,697 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:43,697 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:43,697 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:43,697 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:43,697 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:43,700 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,700 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,700 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,701 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,702 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,708 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:43,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:43,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:43,708 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,709 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:43,708 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,709 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,803 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=585, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/30502fe7231841f0831b808fdbaf57ff 2023-04-23 05:55:43,809 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:43,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:43,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:43,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:43,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:43,814 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:43,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:43,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:43,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,815 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:43,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:43,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:43,817 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:43,817 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,817 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:43,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/ece3abd406f14877b1f9ad1ec122bf2e 2023-04-23 05:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:43,824 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,824 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:43,824 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:43,824 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:43,824 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:43,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,826 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,826 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,826 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:43,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:43,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2de5e7af[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:43,828 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:43,828 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229403819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,830 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:43,830 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:43,830 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:43,830 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,830 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:43,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:43,834 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:43,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:43,841 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:43,841 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:43,841 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:43,841 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:43,841 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:43,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229403835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,842 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,842 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,842 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229403835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:43,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229403835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,843 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,844 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,849 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,849 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,850 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:43,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:43,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,850 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:43,850 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:43,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:43,950 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:43,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:43,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:43,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:43,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:43,955 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:43,955 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,955 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:43,955 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:43,955 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,955 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,956 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,956 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,957 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,957 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:43,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:43,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:43,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:43,958 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:43,958 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:43,958 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:43,959 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,959 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:43,959 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:43,959 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:43,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:43,962 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,962 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:43,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6507c163[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:43,965 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:43,965 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,966 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:43,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:43,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:43,967 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:43,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:43,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,969 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:43,969 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:43,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:43,976 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:43,976 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:43,976 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:43,976 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:43,976 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:43,977 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,977 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,977 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,978 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:43,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:43,979 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:43,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:43,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:43,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:43,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:43,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:43,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:43,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,988 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:43,988 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:43,988 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,988 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:43,988 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:43,988 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:43,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:43,988 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:43,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,088 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:44,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:44,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:44,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:44,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:44,095 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:44,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:44,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:44,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,096 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,096 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,097 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,097 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:44,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:44,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:44,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:44,098 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:44,098 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,098 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:44,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:44,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:44,103 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,103 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:44,103 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:44,103 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:44,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:44,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:44,103 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:44,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,105 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,105 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,105 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:44,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd8d34c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:44,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:44,107 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,108 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:44,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:44,108 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:44,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:44,110 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:44,110 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:44,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:44,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:44,113 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:44,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:44,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:44,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:44,113 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:44,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:44,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:44,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:44,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,121 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:44,121 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:44,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:44,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229404130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229404144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229404144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229404146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,221 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:44,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:44,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:44,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:44,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:44,225 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:44,225 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,225 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:44,225 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:44,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,229 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,229 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:44,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:44,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:44,230 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:44,230 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,230 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:44,236 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,236 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:44,236 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:44,236 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:44,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a90f506[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:44,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:44,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,240 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:44,240 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:44,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:44,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:44,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,242 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:44,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,242 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,242 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:44,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:44,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:44,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:44,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:44,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:44,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:44,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:44,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,248 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,249 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:44,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:44,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:44,251 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,252 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,252 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:44,252 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,252 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:44,252 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=585 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/41d413001f7e489993404175adbd75b0 2023-04-23 05:55:44,258 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/30502fe7231841f0831b808fdbaf57ff as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/30502fe7231841f0831b808fdbaf57ff 2023-04-23 05:55:44,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/30502fe7231841f0831b808fdbaf57ff, entries=350, sequenceid=585, filesize=52.9 K 2023-04-23 05:55:44,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/ece3abd406f14877b1f9ad1ec122bf2e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ece3abd406f14877b1f9ad1ec122bf2e 2023-04-23 05:55:44,267 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ece3abd406f14877b1f9ad1ec122bf2e, entries=150, sequenceid=585, filesize=11.9 K 2023-04-23 05:55:44,267 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/41d413001f7e489993404175adbd75b0 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/41d413001f7e489993404175adbd75b0 2023-04-23 05:55:44,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/41d413001f7e489993404175adbd75b0, entries=150, sequenceid=585, filesize=11.9 K 2023-04-23 05:55:44,271 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 894ms, sequenceid=585, compaction requested=true 2023-04-23 05:55:44,271 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:44,271 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:44,271 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:44,271 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:44,271 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:44,272 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:44,272 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:44,272 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:44,272 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/A is initiating minor compaction (all files) 2023-04-23 05:55:44,272 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/B is initiating minor compaction (all files) 2023-04-23 05:55:44,272 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/A in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,272 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/B in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,272 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4ab30ac1d7e840e4abea019aa50fd758, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f772062291014a4499c5cc6dfd0845df, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/30502fe7231841f0831b808fdbaf57ff] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=104.5 K 2023-04-23 05:55:44,272 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/d7842a2f4aab40f0827e14bf5eedbeca, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/4aa76702cf314de2a3a9dda853539ed2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ece3abd406f14877b1f9ad1ec122bf2e] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=36.4 K 2023-04-23 05:55:44,273 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 4ab30ac1d7e840e4abea019aa50fd758, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1682229340802 2023-04-23 05:55:44,273 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d7842a2f4aab40f0827e14bf5eedbeca, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1682229340802 2023-04-23 05:55:44,273 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting f772062291014a4499c5cc6dfd0845df, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1682229341139 2023-04-23 05:55:44,273 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 4aa76702cf314de2a3a9dda853539ed2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1682229341139 2023-04-23 05:55:44,273 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 30502fe7231841f0831b808fdbaf57ff, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1682229343327 2023-04-23 05:55:44,274 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ece3abd406f14877b1f9ad1ec122bf2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1682229343331 2023-04-23 05:55:44,303 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#B#compaction#371 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:44,306 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#A#compaction#370 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:44,325 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/a87775c2160b43eda44c3751c26b074f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a87775c2160b43eda44c3751c26b074f 2023-04-23 05:55:44,329 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/B of 9b5a61538d9e2f8b71e356a41d6c31f2 into a87775c2160b43eda44c3751c26b074f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:44,329 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:44,329 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/B, priority=13, startTime=1682229344271; duration=0sec 2023-04-23 05:55:44,329 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:44,329 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:44,330 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:44,331 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/C is initiating minor compaction (all files) 2023-04-23 05:55:44,331 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/C in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,331 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b4a6105d2ce04d18b79711578d12613f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/0f5552dac68b4e8090aad42e6b7cb70f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/41d413001f7e489993404175adbd75b0] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=36.4 K 2023-04-23 05:55:44,331 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b4a6105d2ce04d18b79711578d12613f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1682229340802 2023-04-23 05:55:44,331 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 0f5552dac68b4e8090aad42e6b7cb70f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=571, earliestPutTs=1682229341139 2023-04-23 05:55:44,332 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 41d413001f7e489993404175adbd75b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1682229343331 2023-04-23 05:55:44,338 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#C#compaction#372 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 53.85 MB/second 2023-04-23 05:55:44,342 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/c56d4c6e3fbb41508b974f17e2324689 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/c56d4c6e3fbb41508b974f17e2324689 2023-04-23 05:55:44,347 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/A of 9b5a61538d9e2f8b71e356a41d6c31f2 into c56d4c6e3fbb41508b974f17e2324689(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:44,347 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:44,347 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/A, priority=13, startTime=1682229344271; duration=0sec 2023-04-23 05:55:44,347 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:44,352 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:44,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:44,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:44,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:44,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:44,355 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:44,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:44,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:44,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,356 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,357 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:44,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:44,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:44,358 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:44,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:44,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:44,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:44,360 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:44,360 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,360 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:44,360 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=136.42 KB heapSize=384.53 KB 2023-04-23 05:55:44,360 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:44,360 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:44,360 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:44,361 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:44,361 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:44,361 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:44,398 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042335d61b2717be4517bc2a326e25a88223 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042335d61b2717be4517bc2a326e25a88223 2023-04-23 05:55:44,417 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=640, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/9e355878148f4d12a0bb93ecff68f0fd 2023-04-23 05:55:44,459 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/34f2b923e705423292adea1d66fce984 2023-04-23 05:55:44,470 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/a8eb9a071d8c497facbe961d90d7ef17 2023-04-23 05:55:44,475 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/9e355878148f4d12a0bb93ecff68f0fd as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/9e355878148f4d12a0bb93ecff68f0fd 2023-04-23 05:55:44,478 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/9e355878148f4d12a0bb93ecff68f0fd, entries=150, sequenceid=640, filesize=25.4 K 2023-04-23 05:55:44,479 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/34f2b923e705423292adea1d66fce984 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/34f2b923e705423292adea1d66fce984 2023-04-23 05:55:44,489 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/34f2b923e705423292adea1d66fce984, entries=150, sequenceid=640, filesize=11.9 K 2023-04-23 05:55:44,490 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/a8eb9a071d8c497facbe961d90d7ef17 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a8eb9a071d8c497facbe961d90d7ef17 2023-04-23 05:55:44,494 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a8eb9a071d8c497facbe961d90d7ef17, entries=150, sequenceid=640, filesize=11.9 K 2023-04-23 05:55:44,494 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~384.48 KB/393712, currentSize=0 B/0 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 134ms, sequenceid=640, compaction requested=false 2023-04-23 05:55:44,494 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:44,495 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:44,496 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,496 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@184aa164[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:44,498 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:44,499 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:44,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:44,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:44,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:44,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:44,502 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:44,505 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:44,505 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:44,505 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:44,505 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:44,505 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:44,506 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,507 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,508 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,523 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,523 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,523 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,523 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:44,523 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:44,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:44,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,523 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,524 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,524 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,524 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,524 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:44,524 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:44,624 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:44,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:44,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:44,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:44,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:44,628 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:44,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:44,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:44,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,629 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,632 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,635 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,635 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:44,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:44,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:44,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:44,668 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,668 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:44,668 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:44,668 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:44,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:44,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,670 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,670 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:44,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@439d67d7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:44,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:44,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,673 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:44,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:44,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:44,675 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:44,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:44,678 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:44,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:44,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:44,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:44,678 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:44,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:44,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:44,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,685 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:44,685 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:44,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:44,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:44,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:44,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:44,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:44,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:44,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:44,753 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/a2f1b475375a4699929f20701fae0894 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a2f1b475375a4699929f20701fae0894 2023-04-23 05:55:44,753 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:44,753 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:44,758 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/C of 9b5a61538d9e2f8b71e356a41d6c31f2 into a2f1b475375a4699929f20701fae0894(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:44,758 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:44,758 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/C, priority=13, startTime=1682229344272; duration=0sec 2023-04-23 05:55:44,758 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:44,785 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:44,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:44,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:44,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:44,806 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:44,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:44,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:44,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,808 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,812 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,813 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:44,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:44,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,830 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:44,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:44,834 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,834 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:44,834 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:44,834 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:44,837 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:44,852 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,852 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,853 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/605492c5dd914172b39f085ae4140fac, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/96f9317e2cf4436ba692c71e0ad702ac, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/a8113ec86a9848f8aacbac89cf9d11a7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/2f1942149f7a42e7a215cb671ab09ce6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/df21cae0d61d40f48a8054302a7ae44a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/74304af6d6f84ff9b713def30718e10d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/610e0f9636ce4568a5d1ca5ccf1eaffe, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3e25b8903be5475eb9ef7feff43cb9e8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/866a3b4a7b104e689df8d8280d4e4b92, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/194485a0d1754bf0a4caeaf6d87bc2dc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/b0dada35b8144f3198f971294bbe5ce9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/5feee2ed9a4a4efa995dd718d232fd03, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/6a945eba2125408eb306320e18f0c47f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4c2fccb95fee48b3a7306abd35919cec, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4d90cf47fa6044e1a2a18aaeba7e8bd7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3fbae1027b54441290083a93fd3e20ec, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f4d7ce4158eb4835a08245cb373add2a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/8cd5ee94c98d4e78aa69474cd84f883b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/fb08084ef70f4e0b9e35b6c385522c32, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/e6759161155f44cf8076606b98418ee2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/551aabe864d84d4ab48096fc066b81de, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/368e090aeb134245ab0e3962cbca76b2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4ab30ac1d7e840e4abea019aa50fd758, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f772062291014a4499c5cc6dfd0845df, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/30502fe7231841f0831b808fdbaf57ff] to archive 2023-04-23 05:55:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,854 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:55:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:44,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49559e5b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:44,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:44,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,856 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/605492c5dd914172b39f085ae4140fac to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/605492c5dd914172b39f085ae4140fac 2023-04-23 05:55:44,856 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:44,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:44,858 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/96f9317e2cf4436ba692c71e0ad702ac to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/96f9317e2cf4436ba692c71e0ad702ac 2023-04-23 05:55:44,858 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:44,858 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,858 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:44,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,859 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/a8113ec86a9848f8aacbac89cf9d11a7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/a8113ec86a9848f8aacbac89cf9d11a7 2023-04-23 05:55:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,862 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/2f1942149f7a42e7a215cb671ab09ce6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/2f1942149f7a42e7a215cb671ab09ce6 2023-04-23 05:55:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:44,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:44,862 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:44,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:44,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:44,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:44,863 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:44,863 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/df21cae0d61d40f48a8054302a7ae44a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/df21cae0d61d40f48a8054302a7ae44a 2023-04-23 05:55:44,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,865 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,867 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,869 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/74304af6d6f84ff9b713def30718e10d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/74304af6d6f84ff9b713def30718e10d 2023-04-23 05:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,871 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/610e0f9636ce4568a5d1ca5ccf1eaffe to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/610e0f9636ce4568a5d1ca5ccf1eaffe 2023-04-23 05:55:44,872 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3e25b8903be5475eb9ef7feff43cb9e8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3e25b8903be5475eb9ef7feff43cb9e8 2023-04-23 05:55:44,873 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/866a3b4a7b104e689df8d8280d4e4b92 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/866a3b4a7b104e689df8d8280d4e4b92 2023-04-23 05:55:44,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,875 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/194485a0d1754bf0a4caeaf6d87bc2dc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/194485a0d1754bf0a4caeaf6d87bc2dc 2023-04-23 05:55:44,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:44,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229344858, max timestamp=1682229344873Num uniques 859; 2023-04-23 05:55:44,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229404868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:44,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229344858, max timestamp=1682229344873Num uniques 859; 2023-04-23 05:55:44,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:44,876 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/b0dada35b8144f3198f971294bbe5ce9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/b0dada35b8144f3198f971294bbe5ce9 2023-04-23 05:55:44,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229344858, max timestamp=1682229344873Num uniques 859; 2023-04-23 05:55:44,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:44,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:44,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:44,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,878 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:44,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:44,878 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/5feee2ed9a4a4efa995dd718d232fd03 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/5feee2ed9a4a4efa995dd718d232fd03 2023-04-23 05:55:44,878 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:44,879 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/6a945eba2125408eb306320e18f0c47f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/6a945eba2125408eb306320e18f0c47f 2023-04-23 05:55:44,880 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4c2fccb95fee48b3a7306abd35919cec to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4c2fccb95fee48b3a7306abd35919cec 2023-04-23 05:55:44,881 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4d90cf47fa6044e1a2a18aaeba7e8bd7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4d90cf47fa6044e1a2a18aaeba7e8bd7 2023-04-23 05:55:44,883 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3fbae1027b54441290083a93fd3e20ec to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/3fbae1027b54441290083a93fd3e20ec 2023-04-23 05:55:44,884 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f4d7ce4158eb4835a08245cb373add2a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f4d7ce4158eb4835a08245cb373add2a 2023-04-23 05:55:44,885 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/8cd5ee94c98d4e78aa69474cd84f883b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/8cd5ee94c98d4e78aa69474cd84f883b 2023-04-23 05:55:44,886 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/fb08084ef70f4e0b9e35b6c385522c32 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/fb08084ef70f4e0b9e35b6c385522c32 2023-04-23 05:55:44,888 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/e6759161155f44cf8076606b98418ee2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/e6759161155f44cf8076606b98418ee2 2023-04-23 05:55:44,889 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/551aabe864d84d4ab48096fc066b81de to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/551aabe864d84d4ab48096fc066b81de 2023-04-23 05:55:44,890 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/368e090aeb134245ab0e3962cbca76b2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/368e090aeb134245ab0e3962cbca76b2 2023-04-23 05:55:44,892 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4ab30ac1d7e840e4abea019aa50fd758 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/4ab30ac1d7e840e4abea019aa50fd758 2023-04-23 05:55:44,893 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f772062291014a4499c5cc6dfd0845df to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/f772062291014a4499c5cc6dfd0845df 2023-04-23 05:55:44,895 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/30502fe7231841f0831b808fdbaf57ff to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/30502fe7231841f0831b808fdbaf57ff 2023-04-23 05:55:44,902 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e3d7687c82fb45819ad732fdbf6e8287, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a4f9a50da80d46ff947c01e306df2999, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/db58e93fa9fc456782ba487bddab2af2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e0a922bb658f441cac776f4bb979488c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/96b1e64c73454bec830961c786296315, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/171576687bb5460b88479d976c078435, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e9e4cc4971d84c5b90fe3953031f5d82, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/0a1f7a2e98f642c0bac8bc18e23f54a9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/5ccb66a1cb8a4644bb0c303ae93e81dc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/214c1c85bad4475781962be19f78b8c9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ec6ef455d7954a129e19f832788a59c7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/388003a0a0264281961981bb80129dcd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/fd57802a86274068a5fcb78731c42189, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/092520495efe49bcba02acf6df4c0bda, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/74841aacfab547d4bc81ddaadb532ff9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b5bfe1e46a574c4db61ddd67f273b943, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b41415a0183c4cbb98dd90d09b84c61d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e1bdf4434f9c489dacafd38d59a94841, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/698fd6c1d64d407a92c60d3f76e26524, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/7de09abd00fa43c0a82a2fa88d1c7652, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/9518eb2e7b464acc8d3b729c66e8c000, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/d7842a2f4aab40f0827e14bf5eedbeca, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/bd97c410e8d64de499dd6559df0fb806, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/4aa76702cf314de2a3a9dda853539ed2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ece3abd406f14877b1f9ad1ec122bf2e] to archive 2023-04-23 05:55:44,903 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:55:44,905 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e3d7687c82fb45819ad732fdbf6e8287 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e3d7687c82fb45819ad732fdbf6e8287 2023-04-23 05:55:44,906 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a4f9a50da80d46ff947c01e306df2999 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a4f9a50da80d46ff947c01e306df2999 2023-04-23 05:55:44,907 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/db58e93fa9fc456782ba487bddab2af2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/db58e93fa9fc456782ba487bddab2af2 2023-04-23 05:55:44,908 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e0a922bb658f441cac776f4bb979488c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e0a922bb658f441cac776f4bb979488c 2023-04-23 05:55:44,909 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/96b1e64c73454bec830961c786296315 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/96b1e64c73454bec830961c786296315 2023-04-23 05:55:44,910 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/171576687bb5460b88479d976c078435 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/171576687bb5460b88479d976c078435 2023-04-23 05:55:44,912 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e9e4cc4971d84c5b90fe3953031f5d82 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e9e4cc4971d84c5b90fe3953031f5d82 2023-04-23 05:55:44,914 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/0a1f7a2e98f642c0bac8bc18e23f54a9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/0a1f7a2e98f642c0bac8bc18e23f54a9 2023-04-23 05:55:44,916 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/5ccb66a1cb8a4644bb0c303ae93e81dc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/5ccb66a1cb8a4644bb0c303ae93e81dc 2023-04-23 05:55:44,917 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/214c1c85bad4475781962be19f78b8c9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/214c1c85bad4475781962be19f78b8c9 2023-04-23 05:55:44,918 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ec6ef455d7954a129e19f832788a59c7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ec6ef455d7954a129e19f832788a59c7 2023-04-23 05:55:44,920 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/388003a0a0264281961981bb80129dcd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/388003a0a0264281961981bb80129dcd 2023-04-23 05:55:44,921 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/fd57802a86274068a5fcb78731c42189 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/fd57802a86274068a5fcb78731c42189 2023-04-23 05:55:44,922 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/092520495efe49bcba02acf6df4c0bda to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/092520495efe49bcba02acf6df4c0bda 2023-04-23 05:55:44,924 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/74841aacfab547d4bc81ddaadb532ff9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/74841aacfab547d4bc81ddaadb532ff9 2023-04-23 05:55:44,925 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b5bfe1e46a574c4db61ddd67f273b943 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b5bfe1e46a574c4db61ddd67f273b943 2023-04-23 05:55:44,926 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b41415a0183c4cbb98dd90d09b84c61d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/b41415a0183c4cbb98dd90d09b84c61d 2023-04-23 05:55:44,927 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e1bdf4434f9c489dacafd38d59a94841 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/e1bdf4434f9c489dacafd38d59a94841 2023-04-23 05:55:44,928 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/698fd6c1d64d407a92c60d3f76e26524 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/698fd6c1d64d407a92c60d3f76e26524 2023-04-23 05:55:44,929 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/7de09abd00fa43c0a82a2fa88d1c7652 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/7de09abd00fa43c0a82a2fa88d1c7652 2023-04-23 05:55:44,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:44,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229404921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,931 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/9518eb2e7b464acc8d3b729c66e8c000 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/9518eb2e7b464acc8d3b729c66e8c000 2023-04-23 05:55:44,932 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/d7842a2f4aab40f0827e14bf5eedbeca to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/d7842a2f4aab40f0827e14bf5eedbeca 2023-04-23 05:55:44,933 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/bd97c410e8d64de499dd6559df0fb806 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/bd97c410e8d64de499dd6559df0fb806 2023-04-23 05:55:44,934 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/4aa76702cf314de2a3a9dda853539ed2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/4aa76702cf314de2a3a9dda853539ed2 2023-04-23 05:55:44,935 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ece3abd406f14877b1f9ad1ec122bf2e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/ece3abd406f14877b1f9ad1ec122bf2e 2023-04-23 05:55:44,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:44,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229404930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:44,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229404930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,940 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a70fe58f320e490c80153a82ea168430, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/47a13dc1da484c1c9615159c3ae26138, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7ac04993f7d24e80b2a37539996833ef, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/75896ffdb3b34ad3b0c0a32c796acbf8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c303dc021f9c4608961733237d0c60c1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7d278b255af04de5990e72a626d4c4dc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/ac9d1aaeb71f44cb86c2d79e12db87bc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/652182f73a9e457d9df6371dbffd9817, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/6b484b8cb98e475faac256571281b2da, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b3af449c0c8a4ce4b5fb166d8603422e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4e5e63e9424e491a806f86253728246c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4a55b16e282f4442840ad01d0fd03e55, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/442b1b8452c14a6297adf8d4b3b5b49b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/be37cc80b4ee478e9e3b483c7e987171, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/25709f65c44740c0bf7b92d32cfeea79, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9a7110a59d34488d8d7acf91045a4798, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c3ef007ed6d146338da8e83566c9969f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9c465361489d46bd9f15051f2a1ffcad, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/974f0ab7448f4df284cbf96eda7b7cde, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7038173c36f34afca3cac6f2ee5ffc89, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/649eddd244dc453984fa082dabafaf18, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b4a6105d2ce04d18b79711578d12613f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/e619f912a36a4fe0912db2ac9f41abef, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/0f5552dac68b4e8090aad42e6b7cb70f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/41d413001f7e489993404175adbd75b0] to archive 2023-04-23 05:55:44,940 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:55:44,941 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a70fe58f320e490c80153a82ea168430 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a70fe58f320e490c80153a82ea168430 2023-04-23 05:55:44,942 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/47a13dc1da484c1c9615159c3ae26138 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/47a13dc1da484c1c9615159c3ae26138 2023-04-23 05:55:44,943 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7ac04993f7d24e80b2a37539996833ef to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7ac04993f7d24e80b2a37539996833ef 2023-04-23 05:55:44,944 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/75896ffdb3b34ad3b0c0a32c796acbf8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/75896ffdb3b34ad3b0c0a32c796acbf8 2023-04-23 05:55:44,946 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c303dc021f9c4608961733237d0c60c1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c303dc021f9c4608961733237d0c60c1 2023-04-23 05:55:44,947 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7d278b255af04de5990e72a626d4c4dc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7d278b255af04de5990e72a626d4c4dc 2023-04-23 05:55:44,947 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/ac9d1aaeb71f44cb86c2d79e12db87bc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/ac9d1aaeb71f44cb86c2d79e12db87bc 2023-04-23 05:55:44,949 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/652182f73a9e457d9df6371dbffd9817 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/652182f73a9e457d9df6371dbffd9817 2023-04-23 05:55:44,950 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/6b484b8cb98e475faac256571281b2da to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/6b484b8cb98e475faac256571281b2da 2023-04-23 05:55:44,951 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b3af449c0c8a4ce4b5fb166d8603422e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b3af449c0c8a4ce4b5fb166d8603422e 2023-04-23 05:55:44,951 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4e5e63e9424e491a806f86253728246c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4e5e63e9424e491a806f86253728246c 2023-04-23 05:55:44,953 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4a55b16e282f4442840ad01d0fd03e55 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/4a55b16e282f4442840ad01d0fd03e55 2023-04-23 05:55:44,955 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/442b1b8452c14a6297adf8d4b3b5b49b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/442b1b8452c14a6297adf8d4b3b5b49b 2023-04-23 05:55:44,956 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/be37cc80b4ee478e9e3b483c7e987171 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/be37cc80b4ee478e9e3b483c7e987171 2023-04-23 05:55:44,958 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/25709f65c44740c0bf7b92d32cfeea79 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/25709f65c44740c0bf7b92d32cfeea79 2023-04-23 05:55:44,959 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9a7110a59d34488d8d7acf91045a4798 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9a7110a59d34488d8d7acf91045a4798 2023-04-23 05:55:44,960 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c3ef007ed6d146338da8e83566c9969f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/c3ef007ed6d146338da8e83566c9969f 2023-04-23 05:55:44,961 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9c465361489d46bd9f15051f2a1ffcad to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/9c465361489d46bd9f15051f2a1ffcad 2023-04-23 05:55:44,962 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/974f0ab7448f4df284cbf96eda7b7cde to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/974f0ab7448f4df284cbf96eda7b7cde 2023-04-23 05:55:44,962 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7038173c36f34afca3cac6f2ee5ffc89 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/7038173c36f34afca3cac6f2ee5ffc89 2023-04-23 05:55:44,965 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/649eddd244dc453984fa082dabafaf18 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/649eddd244dc453984fa082dabafaf18 2023-04-23 05:55:44,967 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b4a6105d2ce04d18b79711578d12613f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/b4a6105d2ce04d18b79711578d12613f 2023-04-23 05:55:44,967 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/e619f912a36a4fe0912db2ac9f41abef to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/e619f912a36a4fe0912db2ac9f41abef 2023-04-23 05:55:44,968 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/0f5552dac68b4e8090aad42e6b7cb70f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/0f5552dac68b4e8090aad42e6b7cb70f 2023-04-23 05:55:44,969 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/41d413001f7e489993404175adbd75b0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/41d413001f7e489993404175adbd75b0 2023-04-23 05:55:44,978 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:44,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:44,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:44,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:44,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:44,981 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:44,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:44,981 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:44,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,983 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:44,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:44,983 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,984 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,984 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:44,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:44,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:44,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:44,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:44,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. started... 2023-04-23 05:55:44,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. as already flushing 2023-04-23 05:55:44,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:44,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:44,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229404977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:44,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:44,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54707ab2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:44,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:44,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:44,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:44,992 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:44,994 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:44,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:44,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:44,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:44,994 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:44,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:44,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:44,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,998 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:44,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:45,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:45,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:45,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:45,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:45,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:45,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:45,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:45,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:45,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:45,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:45,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:45,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:45,001 DEBUG [Thread-2252] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:45,001 DEBUG [Thread-2252] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:45,006 DEBUG [Thread-2257] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20d28b68 to 127.0.0.1:63802 2023-04-23 05:55:45,006 DEBUG [Thread-2257] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:45,009 DEBUG [Thread-2255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x313c7ad6 to 127.0.0.1:63802 2023-04-23 05:55:45,009 DEBUG [Thread-2255] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:45,010 DEBUG [Thread-2253] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d4bfeca to 127.0.0.1:63802 2023-04-23 05:55:45,010 DEBUG [Thread-2253] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:45,014 DEBUG [Thread-2261] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27313b38 to 127.0.0.1:63802 2023-04-23 05:55:45,014 DEBUG [Thread-2261] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:45,021 DEBUG [Thread-2259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x750d2a7d to 127.0.0.1:63802 2023-04-23 05:55:45,021 DEBUG [Thread-2259] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:45,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229405032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229405038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229405040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,101 DEBUG [Thread-2252] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:45,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:45,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229405188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,205 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304236769f212361745bd81993077d7eaedd1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236769f212361745bd81993077d7eaedd1 2023-04-23 05:55:45,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229405234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229405241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229405242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229405490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229405537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229405543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229405544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:45,615 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=652, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/1d16d5042cc5443fa9262d9e955f7c79 2023-04-23 05:55:45,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:45,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57282 deadline: 1682229405992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:46,027 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/2bf635fd909c40718757e587cf5d9a64 2023-04-23 05:55:46,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:46,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57298 deadline: 1682229406041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:46,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:46,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57272 deadline: 1682229406044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:46,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:46,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57292 deadline: 1682229406047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:46,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/28a23493a4a04c22866eddb984c4db80 2023-04-23 05:55:46,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/1d16d5042cc5443fa9262d9e955f7c79 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/1d16d5042cc5443fa9262d9e955f7c79 2023-04-23 05:55:46,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/1d16d5042cc5443fa9262d9e955f7c79, entries=400, sequenceid=652, filesize=59.8 K 2023-04-23 05:55:46,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/2bf635fd909c40718757e587cf5d9a64 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/2bf635fd909c40718757e587cf5d9a64 2023-04-23 05:55:46,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/2bf635fd909c40718757e587cf5d9a64, entries=150, sequenceid=652, filesize=11.9 K 2023-04-23 05:55:46,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/28a23493a4a04c22866eddb984c4db80 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/28a23493a4a04c22866eddb984c4db80 2023-04-23 05:55:46,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/28a23493a4a04c22866eddb984c4db80, entries=150, sequenceid=652, filesize=11.9 K 2023-04-23 05:55:46,454 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 1702ms, sequenceid=652, compaction requested=true 2023-04-23 05:55:46,454 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:46,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:46,455 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:46,455 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:46,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:46,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:46,456 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 114198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:46,456 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/A is initiating minor compaction (all files) 2023-04-23 05:55:46,456 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:46,456 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/A in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:46,456 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/B is initiating minor compaction (all files) 2023-04-23 05:55:46,456 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/B in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:46,456 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/c56d4c6e3fbb41508b974f17e2324689, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/9e355878148f4d12a0bb93ecff68f0fd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/1d16d5042cc5443fa9262d9e955f7c79] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=111.5 K 2023-04-23 05:55:46,456 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a87775c2160b43eda44c3751c26b074f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/34f2b923e705423292adea1d66fce984, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/2bf635fd909c40718757e587cf5d9a64] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=36.5 K 2023-04-23 05:55:46,456 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c56d4c6e3fbb41508b974f17e2324689, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1682229343331 2023-04-23 05:55:46,456 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a87775c2160b43eda44c3751c26b074f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1682229343331 2023-04-23 05:55:46,457 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 9e355878148f4d12a0bb93ecff68f0fd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229343491 2023-04-23 05:55:46,457 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 34f2b923e705423292adea1d66fce984, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229343491 2023-04-23 05:55:46,457 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 1d16d5042cc5443fa9262d9e955f7c79, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1682229344719 2023-04-23 05:55:46,457 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2bf635fd909c40718757e587cf5d9a64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1682229344748 2023-04-23 05:55:46,464 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#B#compaction#380 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:46,469 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#A#compaction#379 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:46,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:46,601 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-23 05:55:46,601 DEBUG [Thread-2250] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x722b2407 to 127.0.0.1:63802 2023-04-23 05:55:46,601 DEBUG [Thread-2250] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:46,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:46,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:46,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:46,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:46,602 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:46,602 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:46,885 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/adc468b9e9b6496991b09593742f971b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/adc468b9e9b6496991b09593742f971b 2023-04-23 05:55:46,889 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/B of 9b5a61538d9e2f8b71e356a41d6c31f2 into adc468b9e9b6496991b09593742f971b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:46,890 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:46,890 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/B, priority=13, startTime=1682229346455; duration=0sec 2023-04-23 05:55:46,890 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:46,890 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:46,891 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:46,891 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9b5a61538d9e2f8b71e356a41d6c31f2/C is initiating minor compaction (all files) 2023-04-23 05:55:46,891 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9b5a61538d9e2f8b71e356a41d6c31f2/C in TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:46,891 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a2f1b475375a4699929f20701fae0894, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a8eb9a071d8c497facbe961d90d7ef17, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/28a23493a4a04c22866eddb984c4db80] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp, totalSize=36.5 K 2023-04-23 05:55:46,891 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a2f1b475375a4699929f20701fae0894, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=585, earliestPutTs=1682229343331 2023-04-23 05:55:46,892 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a8eb9a071d8c497facbe961d90d7ef17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229343491 2023-04-23 05:55:46,892 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 28a23493a4a04c22866eddb984c4db80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1682229344748 2023-04-23 05:55:46,898 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9b5a61538d9e2f8b71e356a41d6c31f2#C#compaction#382 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:46,906 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/721dfc1aac294835bd243da5bb5111b5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/721dfc1aac294835bd243da5bb5111b5 2023-04-23 05:55:46,910 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/C of 9b5a61538d9e2f8b71e356a41d6c31f2 into 721dfc1aac294835bd243da5bb5111b5(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:46,911 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:46,911 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/C, priority=13, startTime=1682229346455; duration=0sec 2023-04-23 05:55:46,911 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:47,002 DEBUG [Thread-2246] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14e151fb to 127.0.0.1:63802 2023-04-23 05:55:47,002 DEBUG [Thread-2246] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:47,017 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423c26cd30357564be6aab4f53badc892f5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c26cd30357564be6aab4f53badc892f5 2023-04-23 05:55:47,023 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=690, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/84c746d506dd49ce98eb0dcec18be962 2023-04-23 05:55:47,033 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/c831ac2845394741833aa2b3173482df 2023-04-23 05:55:47,048 DEBUG [Thread-2242] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1cee66b5 to 127.0.0.1:63802 2023-04-23 05:55:47,048 DEBUG [Thread-2244] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c1d19b3 to 127.0.0.1:63802 2023-04-23 05:55:47,048 DEBUG [Thread-2244] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:47,048 DEBUG [Thread-2242] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:47,050 DEBUG [Thread-2248] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x455a88b1 to 127.0.0.1:63802 2023-04-23 05:55:47,050 DEBUG [Thread-2248] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:47,050 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-23 05:55:47,050 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-23 05:55:47,050 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 143 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 143 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 158 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 1677 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 5031 rows 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 1673 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 5019 rows 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 1685 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 5055 rows 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 1668 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 5004 rows 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 1674 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 5022 rows 2023-04-23 05:55:47,051 INFO [Listener at localhost/41845] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-23 05:55:47,051 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67b6d133 to 127.0.0.1:63802 2023-04-23 05:55:47,052 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:55:47,054 INFO [Listener at localhost/41845] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-23 05:55:47,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-23 05:55:47,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:47,061 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229347061"}]},"ts":"1682229347061"} 2023-04-23 05:55:47,062 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-23 05:55:47,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-23 05:55:47,157 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-23 05:55:47,158 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, UNASSIGN}] 2023-04-23 05:55:47,160 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, UNASSIGN 2023-04-23 05:55:47,160 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=9b5a61538d9e2f8b71e356a41d6c31f2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:47,161 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229347160"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229347160"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229347160"}]},"ts":"1682229347160"} 2023-04-23 05:55:47,162 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:55:47,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-23 05:55:47,296 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/ba5a50309adf4ad2a4a7145c5f6fe602 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/ba5a50309adf4ad2a4a7145c5f6fe602 2023-04-23 05:55:47,301 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9b5a61538d9e2f8b71e356a41d6c31f2/A of 9b5a61538d9e2f8b71e356a41d6c31f2 into ba5a50309adf4ad2a4a7145c5f6fe602(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:47,301 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:47,301 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2., storeName=9b5a61538d9e2f8b71e356a41d6c31f2/A, priority=13, startTime=1682229346455; duration=0sec 2023-04-23 05:55:47,301 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:47,313 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:47,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9b5a61538d9e2f8b71e356a41d6c31f2, disabling compactions & flushes 2023-04-23 05:55:47,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:47,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-23 05:55:47,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/73cd5e344e834de0a6b3903ae5d23d98 2023-04-23 05:55:47,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/84c746d506dd49ce98eb0dcec18be962 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/84c746d506dd49ce98eb0dcec18be962 2023-04-23 05:55:47,460 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/84c746d506dd49ce98eb0dcec18be962, entries=150, sequenceid=690, filesize=25.4 K 2023-04-23 05:55:47,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/c831ac2845394741833aa2b3173482df as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/c831ac2845394741833aa2b3173482df 2023-04-23 05:55:47,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/c831ac2845394741833aa2b3173482df, entries=150, sequenceid=690, filesize=11.9 K 2023-04-23 05:55:47,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/73cd5e344e834de0a6b3903ae5d23d98 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/73cd5e344e834de0a6b3903ae5d23d98 2023-04-23 05:55:47,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/73cd5e344e834de0a6b3903ae5d23d98, entries=150, sequenceid=690, filesize=11.9 K 2023-04-23 05:55:47,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=26.84 KB/27480 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 872ms, sequenceid=690, compaction requested=false 2023-04-23 05:55:47,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:47,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:47,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:47,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. after waiting 0 ms 2023-04-23 05:55:47,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:47,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9b5a61538d9e2f8b71e356a41d6c31f2 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-23 05:55:47,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=A 2023-04-23 05:55:47,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:47,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=B 2023-04-23 05:55:47,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:47,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9b5a61538d9e2f8b71e356a41d6c31f2, store=C 2023-04-23 05:55:47,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:47,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-23 05:55:47,885 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042398a65eba2dc04e62bf99e40df68c5dd0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042398a65eba2dc04e62bf99e40df68c5dd0 2023-04-23 05:55:48,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-23 05:55:48,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=700, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/20bf7e58a8ad4dab9e6bf43e3bd469b1 2023-04-23 05:55:48,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=700 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/d08efcd6cba24db59b9bdd5c890856c7 2023-04-23 05:55:49,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=700 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/80b8114441874b2fb017e0603fe9a3f7 2023-04-23 05:55:49,116 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/A/20bf7e58a8ad4dab9e6bf43e3bd469b1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/20bf7e58a8ad4dab9e6bf43e3bd469b1 2023-04-23 05:55:49,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/20bf7e58a8ad4dab9e6bf43e3bd469b1, entries=100, sequenceid=700, filesize=18.6 K 2023-04-23 05:55:49,121 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/B/d08efcd6cba24db59b9bdd5c890856c7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/d08efcd6cba24db59b9bdd5c890856c7 2023-04-23 05:55:49,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/d08efcd6cba24db59b9bdd5c890856c7, entries=100, sequenceid=700, filesize=9.5 K 2023-04-23 05:55:49,125 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/.tmp/C/80b8114441874b2fb017e0603fe9a3f7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/80b8114441874b2fb017e0603fe9a3f7 2023-04-23 05:55:49,129 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/80b8114441874b2fb017e0603fe9a3f7, entries=100, sequenceid=700, filesize=9.5 K 2023-04-23 05:55:49,129 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 9b5a61538d9e2f8b71e356a41d6c31f2 in 1656ms, sequenceid=700, compaction requested=true 2023-04-23 05:55:49,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/c56d4c6e3fbb41508b974f17e2324689, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/9e355878148f4d12a0bb93ecff68f0fd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/1d16d5042cc5443fa9262d9e955f7c79] to archive 2023-04-23 05:55:49,130 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:55:49,132 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/c56d4c6e3fbb41508b974f17e2324689 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/c56d4c6e3fbb41508b974f17e2324689 2023-04-23 05:55:49,133 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/9e355878148f4d12a0bb93ecff68f0fd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/9e355878148f4d12a0bb93ecff68f0fd 2023-04-23 05:55:49,134 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/1d16d5042cc5443fa9262d9e955f7c79 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/1d16d5042cc5443fa9262d9e955f7c79 2023-04-23 05:55:49,135 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a87775c2160b43eda44c3751c26b074f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/34f2b923e705423292adea1d66fce984, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/2bf635fd909c40718757e587cf5d9a64] to archive 2023-04-23 05:55:49,135 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:55:49,136 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a87775c2160b43eda44c3751c26b074f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/a87775c2160b43eda44c3751c26b074f 2023-04-23 05:55:49,137 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/34f2b923e705423292adea1d66fce984 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/34f2b923e705423292adea1d66fce984 2023-04-23 05:55:49,138 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/2bf635fd909c40718757e587cf5d9a64 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/2bf635fd909c40718757e587cf5d9a64 2023-04-23 05:55:49,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a2f1b475375a4699929f20701fae0894, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a8eb9a071d8c497facbe961d90d7ef17, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/28a23493a4a04c22866eddb984c4db80] to archive 2023-04-23 05:55:49,139 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:55:49,141 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a2f1b475375a4699929f20701fae0894 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a2f1b475375a4699929f20701fae0894 2023-04-23 05:55:49,141 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a8eb9a071d8c497facbe961d90d7ef17 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/a8eb9a071d8c497facbe961d90d7ef17 2023-04-23 05:55:49,142 DEBUG [StoreCloser-TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/28a23493a4a04c22866eddb984c4db80 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/28a23493a4a04c22866eddb984c4db80 2023-04-23 05:55:49,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/recovered.edits/703.seqid, newMaxSeqId=703, maxSeqId=4 2023-04-23 05:55:49,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2. 2023-04-23 05:55:49,147 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9b5a61538d9e2f8b71e356a41d6c31f2: 2023-04-23 05:55:49,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:49,149 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=9b5a61538d9e2f8b71e356a41d6c31f2, regionState=CLOSED 2023-04-23 05:55:49,150 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229349149"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229349149"}]},"ts":"1682229349149"} 2023-04-23 05:55:49,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-23 05:55:49,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 9b5a61538d9e2f8b71e356a41d6c31f2, server=jenkins-hbase4.apache.org,40995,1682229223026 in 1.9880 sec 2023-04-23 05:55:49,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-23 05:55:49,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9b5a61538d9e2f8b71e356a41d6c31f2, UNASSIGN in 1.9940 sec 2023-04-23 05:55:49,154 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229349154"}]},"ts":"1682229349154"} 2023-04-23 05:55:49,155 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-23 05:55:49,156 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-23 05:55:49,158 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1020 sec 2023-04-23 05:55:49,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-23 05:55:49,169 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-23 05:55:49,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-23 05:55:49,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:49,172 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:49,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-23 05:55:49,173 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:49,177 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:49,180 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/recovered.edits] 2023-04-23 05:55:49,183 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/20bf7e58a8ad4dab9e6bf43e3bd469b1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/20bf7e58a8ad4dab9e6bf43e3bd469b1 2023-04-23 05:55:49,184 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/84c746d506dd49ce98eb0dcec18be962 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/84c746d506dd49ce98eb0dcec18be962 2023-04-23 05:55:49,187 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/ba5a50309adf4ad2a4a7145c5f6fe602 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/A/ba5a50309adf4ad2a4a7145c5f6fe602 2023-04-23 05:55:49,190 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/adc468b9e9b6496991b09593742f971b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/adc468b9e9b6496991b09593742f971b 2023-04-23 05:55:49,191 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/c831ac2845394741833aa2b3173482df to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/c831ac2845394741833aa2b3173482df 2023-04-23 05:55:49,192 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/d08efcd6cba24db59b9bdd5c890856c7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/B/d08efcd6cba24db59b9bdd5c890856c7 2023-04-23 05:55:49,194 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/721dfc1aac294835bd243da5bb5111b5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/721dfc1aac294835bd243da5bb5111b5 2023-04-23 05:55:49,195 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/73cd5e344e834de0a6b3903ae5d23d98 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/73cd5e344e834de0a6b3903ae5d23d98 2023-04-23 05:55:49,197 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/80b8114441874b2fb017e0603fe9a3f7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/C/80b8114441874b2fb017e0603fe9a3f7 2023-04-23 05:55:49,199 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/recovered.edits/703.seqid to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2/recovered.edits/703.seqid 2023-04-23 05:55:49,200 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9b5a61538d9e2f8b71e356a41d6c31f2 2023-04-23 05:55:49,200 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:55:49,200 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-23 05:55:49,201 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-23 05:55:49,204 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042329e1c038904a4252837dcc6f411a2554 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042329e1c038904a4252837dcc6f411a2554 2023-04-23 05:55:49,205 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042335d61b2717be4517bc2a326e25a88223 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042335d61b2717be4517bc2a326e25a88223 2023-04-23 05:55:49,206 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233fc0d333b4d44f39853bd384f674b8de to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233fc0d333b4d44f39853bd384f674b8de 2023-04-23 05:55:49,207 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042344bf2a05b8d4439b993cfafac6c9fe6f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042344bf2a05b8d4439b993cfafac6c9fe6f 2023-04-23 05:55:49,208 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304234a2bdfa6127a4ffc8c9e7100ea6796ba to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304234a2bdfa6127a4ffc8c9e7100ea6796ba 2023-04-23 05:55:49,209 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304234c2e0ef293334e9095da8523356206b2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304234c2e0ef293334e9095da8523356206b2 2023-04-23 05:55:49,210 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304234ec48449c366435f902b8165ce490896 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304234ec48449c366435f902b8165ce490896 2023-04-23 05:55:49,211 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236769f212361745bd81993077d7eaedd1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236769f212361745bd81993077d7eaedd1 2023-04-23 05:55:49,212 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304237c86c5c57ad5408586fe23b14b78d6e5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304237c86c5c57ad5408586fe23b14b78d6e5 2023-04-23 05:55:49,213 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042398a65eba2dc04e62bf99e40df68c5dd0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042398a65eba2dc04e62bf99e40df68c5dd0 2023-04-23 05:55:49,213 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a0f1829933c941cea2d003cda3375ad1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a0f1829933c941cea2d003cda3375ad1 2023-04-23 05:55:49,215 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a6a7f6f2a6d845e28b0331d74f024a6b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a6a7f6f2a6d845e28b0331d74f024a6b 2023-04-23 05:55:49,216 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a9bb134954ca4fdc992c1fc9f06150fa to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a9bb134954ca4fdc992c1fc9f06150fa 2023-04-23 05:55:49,217 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423b67a04f691084f318e07a8d19610ac27 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423b67a04f691084f318e07a8d19610ac27 2023-04-23 05:55:49,217 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c0a4783c21a94b0c9e3dcae3904a003e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c0a4783c21a94b0c9e3dcae3904a003e 2023-04-23 05:55:49,218 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c26cd30357564be6aab4f53badc892f5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c26cd30357564be6aab4f53badc892f5 2023-04-23 05:55:49,219 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c55d34acf71841e7b376f01e542f7470 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c55d34acf71841e7b376f01e542f7470 2023-04-23 05:55:49,220 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c7f7e60a8e734dd09fd8078eb2c1682e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c7f7e60a8e734dd09fd8078eb2c1682e 2023-04-23 05:55:49,221 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423cdcc7aa1a7b04b3cbfa6b15a35c67ca8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423cdcc7aa1a7b04b3cbfa6b15a35c67ca8 2023-04-23 05:55:49,222 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d7ce2cf272624e0bafa4ae6c1886f69a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d7ce2cf272624e0bafa4ae6c1886f69a 2023-04-23 05:55:49,223 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e069761aa78545ca8dc24689f796ed35 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e069761aa78545ca8dc24689f796ed35 2023-04-23 05:55:49,224 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e6e3af0381af4a86acd14fc6e45c2a53 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e6e3af0381af4a86acd14fc6e45c2a53 2023-04-23 05:55:49,225 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f8f4582a82904f8d80a4d50beae51e5f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f8f4582a82904f8d80a4d50beae51e5f 2023-04-23 05:55:49,226 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f92c53ee619042f5adbcfdd3d70fe286 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f92c53ee619042f5adbcfdd3d70fe286 2023-04-23 05:55:49,227 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-23 05:55:49,229 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:49,230 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-23 05:55:49,232 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-23 05:55:49,232 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:49,232 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-23 05:55:49,233 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682229349233"}]},"ts":"9223372036854775807"} 2023-04-23 05:55:49,234 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-23 05:55:49,234 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9b5a61538d9e2f8b71e356a41d6c31f2, NAME => 'TestAcidGuarantees,,1682229321922.9b5a61538d9e2f8b71e356a41d6c31f2.', STARTKEY => '', ENDKEY => ''}] 2023-04-23 05:55:49,234 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-23 05:55:49,234 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682229349234"}]},"ts":"9223372036854775807"} 2023-04-23 05:55:49,235 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-23 05:55:49,237 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:49,238 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-04-23 05:55:49,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-23 05:55:49,274 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-23 05:55:49,288 INFO [Listener at localhost/41845] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=238 (was 226) - Thread LEAK? -, OpenFileDescriptor=569 (was 529) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=384 (was 350) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=12662 (was 12336) - AvailableMemoryMB LEAK? - 2023-04-23 05:55:49,302 INFO [Listener at localhost/41845] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=238, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=384, ProcessCount=169, AvailableMemoryMB=12662 2023-04-23 05:55:49,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-23 05:55:49,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:55:49,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-23 05:55:49,307 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-23 05:55:49,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-23 05:55:49,308 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-23 05:55:49,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-23 05:55:49,309 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:49,310 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d empty. 2023-04-23 05:55:49,310 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:49,310 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:55:49,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-23 05:55:49,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-23 05:55:49,723 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-23 05:55:49,725 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 70eda440525723289f2daf7f6251db0d, NAME => 'TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp 2023-04-23 05:55:49,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-23 05:55:50,134 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:55:50,134 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 70eda440525723289f2daf7f6251db0d, disabling compactions & flushes 2023-04-23 05:55:50,134 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:50,134 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:50,134 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. after waiting 0 ms 2023-04-23 05:55:50,134 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:50,134 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:50,134 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:50,137 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-23 05:55:50,137 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229350137"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229350137"}]},"ts":"1682229350137"} 2023-04-23 05:55:50,138 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-23 05:55:50,139 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-23 05:55:50,139 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229350139"}]},"ts":"1682229350139"} 2023-04-23 05:55:50,140 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-23 05:55:50,142 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=70eda440525723289f2daf7f6251db0d, ASSIGN}] 2023-04-23 05:55:50,144 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=70eda440525723289f2daf7f6251db0d, ASSIGN 2023-04-23 05:55:50,144 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=70eda440525723289f2daf7f6251db0d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=false 2023-04-23 05:55:50,296 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=70eda440525723289f2daf7f6251db0d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:50,296 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229350296"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229350296"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229350296"}]},"ts":"1682229350296"} 2023-04-23 05:55:50,297 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:55:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-23 05:55:50,451 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:50,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 70eda440525723289f2daf7f6251db0d, NAME => 'TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:55:50,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:50,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:55:50,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:50,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:50,453 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:50,454 DEBUG [StoreOpener-70eda440525723289f2daf7f6251db0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A 2023-04-23 05:55:50,454 DEBUG [StoreOpener-70eda440525723289f2daf7f6251db0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A 2023-04-23 05:55:50,454 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:55:50,454 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70eda440525723289f2daf7f6251db0d columnFamilyName A 2023-04-23 05:55:50,455 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] regionserver.HStore(310): Store=70eda440525723289f2daf7f6251db0d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:55:50,455 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:50,456 DEBUG [StoreOpener-70eda440525723289f2daf7f6251db0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B 2023-04-23 05:55:50,456 DEBUG [StoreOpener-70eda440525723289f2daf7f6251db0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B 2023-04-23 05:55:50,456 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:55:50,456 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70eda440525723289f2daf7f6251db0d columnFamilyName B 2023-04-23 05:55:50,457 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] regionserver.HStore(310): Store=70eda440525723289f2daf7f6251db0d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:55:50,457 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:50,458 DEBUG [StoreOpener-70eda440525723289f2daf7f6251db0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C 2023-04-23 05:55:50,458 DEBUG [StoreOpener-70eda440525723289f2daf7f6251db0d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C 2023-04-23 05:55:50,458 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:55:50,458 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70eda440525723289f2daf7f6251db0d columnFamilyName C 2023-04-23 05:55:50,459 INFO [StoreOpener-70eda440525723289f2daf7f6251db0d-1] regionserver.HStore(310): Store=70eda440525723289f2daf7f6251db0d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:55:50,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:50,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:50,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:50,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:55:50,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:50,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-23 05:55:50,465 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 70eda440525723289f2daf7f6251db0d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60079715, jitterRate=-0.10474248230457306}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:55:50,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:50,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., pid=49, masterSystemTime=1682229350449 2023-04-23 05:55:50,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:50,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:50,467 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=70eda440525723289f2daf7f6251db0d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:50,467 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229350467"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229350467"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229350467"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229350467"}]},"ts":"1682229350467"} 2023-04-23 05:55:50,469 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-23 05:55:50,470 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 in 171 msec 2023-04-23 05:55:50,471 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-23 05:55:50,471 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=70eda440525723289f2daf7f6251db0d, ASSIGN in 327 msec 2023-04-23 05:55:50,471 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-23 05:55:50,471 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229350471"}]},"ts":"1682229350471"} 2023-04-23 05:55:50,472 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-23 05:55:50,474 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-23 05:55:50,475 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1690 sec 2023-04-23 05:55:51,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-23 05:55:51,414 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-23 05:55:51,415 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x4539ed0c to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,424 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15840296, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,425 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,428 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38332, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,429 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-23 05:55:51,431 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-23 05:55:51,432 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x32cd672d to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,436 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3816bdf5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,436 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x09b8ff30 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,440 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50150525, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,441 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x4c050db8 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,444 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28a3a77, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,445 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x28e3107b to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,452 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cefb012, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,452 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x14f9adf3 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,456 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24a917df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,456 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x78fa3dee to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,460 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1401cac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,460 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x36d2d826 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,467 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f6e865f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,468 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x6871d7d8 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,475 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48c22118, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,476 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x3ece3076 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,483 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e7ef60, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,483 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x0adb6988 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:55:51,488 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45d9b525, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:55:51,495 DEBUG [hconnection-0x13698dc2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,495 DEBUG [hconnection-0x3bb87b35-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,499 DEBUG [hconnection-0x4fc5effc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:51,496 DEBUG [hconnection-0x2955718c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,499 DEBUG [hconnection-0x6d379892-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,500 DEBUG [hconnection-0x3875b831-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,501 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,502 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38360, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,506 DEBUG [hconnection-0x602daeea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,506 DEBUG [hconnection-0x791b819e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,507 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38380, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,507 DEBUG [hconnection-0x5bf05723-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,510 DEBUG [hconnection-0x12bfa069-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:55:51,510 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,510 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,511 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38368, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,511 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,511 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,520 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38400, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,520 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38404, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:55:51,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:51,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:51,524 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:51,525 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:51,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:51,525 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:51,526 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:51,526 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:51,527 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,527 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:51,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:51,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:51,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:51,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:51,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:51,532 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:51,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:51,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:51,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229411536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229411537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229411537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229411538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:51,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229411544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:51,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:51,552 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:51,552 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:51,552 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:51,552 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:51,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:51,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:51,553 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:51,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:51,557 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,557 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:51,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19fca0d9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:51,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:51,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,561 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:51,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:51,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:51,561 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,562 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:51,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:51,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/270f0fbf9abc4917aa02425f9e2cffda 2023-04-23 05:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:51,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:51,565 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:51,565 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:51,565 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:51,565 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:51,565 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:51,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:51,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,569 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,570 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:51,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:51,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:51,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:51,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:51,573 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:51,573 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:51,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:51,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:51,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229411639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229411639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229411639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229411639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229411645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,673 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:51,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:51,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:51,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:51,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:51,683 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:51,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:51,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:51,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:51,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,684 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:51,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:51,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:51,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:51,695 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:51,700 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:51,700 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:51,700 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:51,700 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:51,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,702 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:51,704 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:51,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f7dfee[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:51,704 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,706 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,706 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:51,706 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:51,707 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:51,707 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,707 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:51,708 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,708 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:51,708 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:51,708 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:51,710 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:51,710 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:51,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:51,710 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:51,711 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:51,711 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:51,712 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,712 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,712 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:51,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,713 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,714 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:51,716 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:51,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:51,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:51,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:51,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:51,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:51,716 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,716 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:51,716 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:51,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:51,817 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:51,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:51,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:51,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:51,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:51,820 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:51,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:51,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:51,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:51,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:51,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:51,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:51,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:51,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:51,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:51,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:51,826 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:51,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:51,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:51,828 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:51,828 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:51,828 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:51,828 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:51,829 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,829 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,830 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:51,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@543528cd[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:51,831 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:51,831 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,832 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:51,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:51,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:51,834 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:51,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:51,839 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:51,839 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:51,839 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:51,839 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:51,839 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:51,841 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,841 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,841 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:51,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:51,841 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,843 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,845 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:51,845 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:51,845 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,845 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:51,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:51,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:51,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:51,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:51,846 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:51,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:51,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,846 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:51,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229411843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229411844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229411845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229411845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:51,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229411850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,947 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:51,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:51,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:51,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:51,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:51,950 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:51,950 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:51,950 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:51,950 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:51,952 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,952 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,953 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:51,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:51,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:51,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:51,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:51,954 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:51,954 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:51,954 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:51,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:51,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:51,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:51,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:51,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:51,959 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,960 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,960 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,960 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:51,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c77d13[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:51,961 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:51,961 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,962 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:51,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:51,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:51,964 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:51,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:51,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:51,964 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:51,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:51,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:51,964 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:51,965 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:51,967 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,967 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,967 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:51,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:51,969 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,971 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:51,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:51,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:51,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:51,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:51,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:51,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:51,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:51,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:51,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:51,975 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:51,975 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:51,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:51,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/ec55f638120f4badb33279145138744e 2023-04-23 05:55:52,075 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:52,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:52,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:52,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:52,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:52,080 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:52,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:52,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:52,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:52,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,082 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,082 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:52,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,082 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,082 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:52,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:52,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:52,083 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:52,084 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:52,084 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:52,084 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:52,084 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:52,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:52,086 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:52,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:52,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc1ffda[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:52,088 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:52,088 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,090 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,090 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,090 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:52,092 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:52,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:52,096 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:52,097 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:52,097 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:52,097 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:52,097 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:52,098 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,098 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,098 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:52,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:52,099 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:52,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,105 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:52,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:52,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:52,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:52,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:52,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:52,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:52,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,105 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:52,106 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:52,105 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:52,106 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229412149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229412151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229412151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229412152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229412158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,206 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:52,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:52,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:52,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:52,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:52,209 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:52,209 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:52,209 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:52,209 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:52,210 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,210 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,211 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,211 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:52,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:52,211 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,211 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:52,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:52,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:52,213 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:52,213 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:52,213 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:52,213 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:52,213 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:52,213 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:52,213 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:52,213 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:52,213 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:52,213 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:52,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:52,216 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:52,216 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:52,216 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,216 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:52,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:52,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70c65763[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:52,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:52,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,219 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:52,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:52,219 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:52,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,219 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:52,220 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:52,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:52,220 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:52,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:52,223 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:52,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:52,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:52,223 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:52,223 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:52,225 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,225 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:52,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,225 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:52,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:52,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,227 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:52,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:52,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:52,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:52,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:52,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:52,231 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:52,231 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:52,231 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:52,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:52,331 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:52,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:52,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:52,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:52,335 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:52,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:52,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:52,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:52,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,337 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:52,337 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:52,337 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,337 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:52,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:52,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:52,338 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:52,338 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:52,338 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:52,339 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:52,339 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:52,339 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:52,339 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:52,339 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:52,339 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:52,339 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:52,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:52,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:52,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:52,341 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:52,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:52,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:52,343 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,343 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:52,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:52,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dc86af6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:52,344 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:52,344 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:52,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:52,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:52,345 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,345 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:52,350 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,350 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:52,350 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:52,350 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:52,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:52,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:52,353 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:52,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:52,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:52,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:52,354 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:52,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:52,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:52,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:52,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:52,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:52,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:52,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:52,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:52,361 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:52,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:52,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:52,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,362 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,362 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,362 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,362 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,362 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:52,362 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:52,362 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:52,362 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:52,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:52,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/5aa3c012815944cc910b01b446a502b7 2023-04-23 05:55:52,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/270f0fbf9abc4917aa02425f9e2cffda as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/270f0fbf9abc4917aa02425f9e2cffda 2023-04-23 05:55:52,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/270f0fbf9abc4917aa02425f9e2cffda, entries=200, sequenceid=14, filesize=13.9 K 2023-04-23 05:55:52,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/ec55f638120f4badb33279145138744e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/ec55f638120f4badb33279145138744e 2023-04-23 05:55:52,420 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/ec55f638120f4badb33279145138744e, entries=150, sequenceid=14, filesize=11.6 K 2023-04-23 05:55:52,421 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/5aa3c012815944cc910b01b446a502b7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5aa3c012815944cc910b01b446a502b7 2023-04-23 05:55:52,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5aa3c012815944cc910b01b446a502b7, entries=150, sequenceid=14, filesize=11.6 K 2023-04-23 05:55:52,426 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 70eda440525723289f2daf7f6251db0d in 900ms, sequenceid=14, compaction requested=false 2023-04-23 05:55:52,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:52,462 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:52,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:52,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:52,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:52,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:52,465 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:52,465 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:52,466 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:52,466 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:52,466 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,466 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,467 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,467 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:52,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:52,468 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,468 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:52,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:52,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:52,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:52,469 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:52,469 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:52,469 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:52,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:52,470 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:52,470 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:52,470 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:55:52,470 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:52,470 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:52,470 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:52,470 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:52,470 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:52,470 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:52,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:52,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:52,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229412667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229412668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229412668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229412673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229412674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229412775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229412775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229412776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229412777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229412778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,883 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/1a316184305845fcb12ba20c4e50e120 2023-04-23 05:55:52,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229412982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229412982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229412983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229412984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:52,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:52,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229412984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,295 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/9388010a918641c5a21c09943750bde2 2023-04-23 05:55:53,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229413291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229413291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229413291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229413291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229413292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,706 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/19947a82b7034b7491ebc79fb936ce5b 2023-04-23 05:55:53,710 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/1a316184305845fcb12ba20c4e50e120 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1a316184305845fcb12ba20c4e50e120 2023-04-23 05:55:53,714 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1a316184305845fcb12ba20c4e50e120, entries=150, sequenceid=34, filesize=11.6 K 2023-04-23 05:55:53,715 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/9388010a918641c5a21c09943750bde2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9388010a918641c5a21c09943750bde2 2023-04-23 05:55:53,719 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9388010a918641c5a21c09943750bde2, entries=150, sequenceid=34, filesize=11.6 K 2023-04-23 05:55:53,719 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/19947a82b7034b7491ebc79fb936ce5b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/19947a82b7034b7491ebc79fb936ce5b 2023-04-23 05:55:53,723 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/19947a82b7034b7491ebc79fb936ce5b, entries=150, sequenceid=34, filesize=11.6 K 2023-04-23 05:55:53,723 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 70eda440525723289f2daf7f6251db0d in 1253ms, sequenceid=34, compaction requested=false 2023-04-23 05:55:53,723 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:53,723 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:53,723 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:53,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:53,726 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,726 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:53,726 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,726 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:53,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:53,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@355fa1b9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:53,728 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:53,728 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,730 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:53,730 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:53,730 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,730 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:53,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,732 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:53,732 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:53,732 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:53,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:53,735 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:53,735 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:53,735 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:53,735 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:53,735 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:53,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:53,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:53,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,737 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:53,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:53,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,742 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,742 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:53,742 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,742 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,742 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:53,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:53,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:53,743 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,743 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,743 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,743 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,743 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:53,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:53,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:53,743 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:53,743 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:53,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:53,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:53,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-23 05:55:53,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:53,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:53,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:53,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:53,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:53,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:53,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/4c28ed8a0e9f4049846908e5a9eaad7b 2023-04-23 05:55:53,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/a3d50961a2cd4c99a382518a2f10dc4f 2023-04-23 05:55:53,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229413823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,843 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:53,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:53,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229413833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229413833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229413834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229413834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:53,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:53,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:53,847 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:53,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:53,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:53,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:53,847 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,848 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,849 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:53,849 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:53,849 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,849 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:53,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:53,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:53,851 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:53,851 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:53,851 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:53,851 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:53,851 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:53,852 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:53,853 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:53,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:53,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4510a7e7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:53,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:53,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,856 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:53,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:53,859 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:53,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:53,875 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:53,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:53,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:53,875 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:53,875 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:53,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,876 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:53,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:53,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:53,877 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:53,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:53,881 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:53,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:53,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:53,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:53,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:53,882 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:53,882 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:53,882 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:53,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:53,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:53,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:53,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229413935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229413945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229413946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229413946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229413946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,982 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:53,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:53,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:53,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:53,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:53,986 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:53,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:53,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:53,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:53,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,986 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,987 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:53,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:53,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,988 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:53,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:53,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:53,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:53,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:53,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:53,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:53,989 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:53,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:53,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:53,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:53,989 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:53,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:53,992 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,992 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:53,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:53,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:53,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72f8d69[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:53,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,995 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:53,995 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:53,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:53,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:53,997 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:53,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:53,999 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:53,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:53,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:53,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:53,999 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:54,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:54,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:54,004 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:54,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:54,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:54,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:54,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:54,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:54,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:54,008 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:54,008 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:54,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:54,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:54,108 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:54,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:54,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:54,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:54,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:54,112 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:54,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:54,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:54,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:54,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:54,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:54,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:54,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:54,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:54,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:54,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:54,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:54,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:54,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:54,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:54,117 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:54,117 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:54,119 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:54,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49fed8a0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:54,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:54,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,123 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:54,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:54,126 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:54,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:54,128 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:54,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:54,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:54,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:54,129 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:54,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:54,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,131 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,134 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,134 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,134 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:54,134 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:54,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:54,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:54,134 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,134 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,134 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,134 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:54,134 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,134 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:54,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:54,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:54,135 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:54,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:54,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229414141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229414152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229414153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229414153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229414154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,235 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:54,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:54,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:54,238 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/ffccb791853d471da705c8b26549ed5f 2023-04-23 05:55:54,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:54,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:54,239 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:54,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:54,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:54,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:54,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,239 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:54,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:54,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:54,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:54,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:54,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:54,244 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,244 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:54,244 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:54,244 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:54,244 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:54,246 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/4c28ed8a0e9f4049846908e5a9eaad7b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/4c28ed8a0e9f4049846908e5a9eaad7b 2023-04-23 05:55:54,246 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:54,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4fa4f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:54,248 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:54,248 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,249 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:54,250 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,250 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:54,250 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/4c28ed8a0e9f4049846908e5a9eaad7b, entries=200, sequenceid=48, filesize=13.9 K 2023-04-23 05:55:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:54,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,251 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,251 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/a3d50961a2cd4c99a382518a2f10dc4f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a3d50961a2cd4c99a382518a2f10dc4f 2023-04-23 05:55:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a3d50961a2cd4c99a382518a2f10dc4f, entries=150, sequenceid=48, filesize=11.6 K 2023-04-23 05:55:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/ffccb791853d471da705c8b26549ed5f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ffccb791853d471da705c8b26549ed5f 2023-04-23 05:55:54,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:54,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:54,256 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:54,256 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:54,256 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:54,256 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:54,256 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:54,259 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,259 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,259 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:54,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ffccb791853d471da705c8b26549ed5f, entries=150, sequenceid=48, filesize=11.6 K 2023-04-23 05:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:54,260 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,260 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 70eda440525723289f2daf7f6251db0d in 458ms, sequenceid=48, compaction requested=true 2023-04-23 05:55:54,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:54,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,261 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:54,261 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:54,261 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:54,261 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,261 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,262 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:54,262 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:55:54,262 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:54,262 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,262 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:55:54,262 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/ec55f638120f4badb33279145138744e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9388010a918641c5a21c09943750bde2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a3d50961a2cd4c99a382518a2f10dc4f] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=34.8 K 2023-04-23 05:55:54,262 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,262 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/270f0fbf9abc4917aa02425f9e2cffda, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1a316184305845fcb12ba20c4e50e120, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/4c28ed8a0e9f4049846908e5a9eaad7b] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=39.4 K 2023-04-23 05:55:54,262 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ec55f638120f4badb33279145138744e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682229351524 2023-04-23 05:55:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,262 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 270f0fbf9abc4917aa02425f9e2cffda, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682229351523 2023-04-23 05:55:54,263 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9388010a918641c5a21c09943750bde2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682229351535 2023-04-23 05:55:54,263 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 1a316184305845fcb12ba20c4e50e120, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682229351535 2023-04-23 05:55:54,263 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a3d50961a2cd4c99a382518a2f10dc4f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229352666 2023-04-23 05:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,264 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 4c28ed8a0e9f4049846908e5a9eaad7b, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229352666 2023-04-23 05:55:54,265 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:54,265 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:54,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:54,265 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:54,265 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:54,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:54,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,266 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:54,266 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:54,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:54,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:54,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:54,270 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#398 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:54,270 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:54,292 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/1d4f0f83ae484153a4fbdf85320be02b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1d4f0f83ae484153a4fbdf85320be02b 2023-04-23 05:55:54,294 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/a7cc7d2a72974a1d92533eb1e25d9e10 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a7cc7d2a72974a1d92533eb1e25d9e10 2023-04-23 05:55:54,298 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into 1d4f0f83ae484153a4fbdf85320be02b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:54,298 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:54,298 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=13, startTime=1682229354261; duration=0sec 2023-04-23 05:55:54,298 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:54,298 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:54,299 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into a7cc7d2a72974a1d92533eb1e25d9e10(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:54,299 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:54,299 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=13, startTime=1682229354260; duration=0sec 2023-04-23 05:55:54,299 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:54,300 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:54,300 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/C is initiating minor compaction (all files) 2023-04-23 05:55:54,300 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/C in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,300 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5aa3c012815944cc910b01b446a502b7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/19947a82b7034b7491ebc79fb936ce5b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ffccb791853d471da705c8b26549ed5f] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=34.8 K 2023-04-23 05:55:54,301 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 5aa3c012815944cc910b01b446a502b7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1682229351524 2023-04-23 05:55:54,302 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 19947a82b7034b7491ebc79fb936ce5b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1682229351535 2023-04-23 05:55:54,302 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ffccb791853d471da705c8b26549ed5f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229352666 2023-04-23 05:55:54,307 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#C#compaction#399 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:54,322 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/e34cbbba0ff7498da8086e175184416c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/e34cbbba0ff7498da8086e175184416c 2023-04-23 05:55:54,326 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/C of 70eda440525723289f2daf7f6251db0d into e34cbbba0ff7498da8086e175184416c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:54,326 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:54,326 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/C, priority=13, startTime=1682229354261; duration=0sec 2023-04-23 05:55:54,327 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:54,366 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:54,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:54,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:54,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:54,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:54,370 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:54,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:54,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:54,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:54,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:54,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:54,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,372 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:54,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:54,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:54,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:54,373 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:54,373 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:54,373 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:54,374 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,374 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:54,374 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-23 05:55:54,374 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:54,374 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:54,374 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:54,374 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:54,374 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:54,374 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:54,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:54,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:54,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229414468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229414469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229414469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229414470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229414470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229414574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229414576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229414576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229414576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229414576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,678 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-23 05:55:54,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229414777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229414779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229414779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229414780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:54,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229414780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,790 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/a80c45db56c04da8a4497f603536427a 2023-04-23 05:55:54,801 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/5a9fd7779a264a28b232c1c0b183d614 2023-04-23 05:55:54,815 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1c59f418dadf466a822241390037cc24 2023-04-23 05:55:54,820 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/a80c45db56c04da8a4497f603536427a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a80c45db56c04da8a4497f603536427a 2023-04-23 05:55:54,825 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a80c45db56c04da8a4497f603536427a, entries=150, sequenceid=69, filesize=11.6 K 2023-04-23 05:55:54,825 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/5a9fd7779a264a28b232c1c0b183d614 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/5a9fd7779a264a28b232c1c0b183d614 2023-04-23 05:55:54,829 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/5a9fd7779a264a28b232c1c0b183d614, entries=150, sequenceid=69, filesize=11.6 K 2023-04-23 05:55:54,830 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1c59f418dadf466a822241390037cc24 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c59f418dadf466a822241390037cc24 2023-04-23 05:55:54,834 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c59f418dadf466a822241390037cc24, entries=150, sequenceid=69, filesize=11.6 K 2023-04-23 05:55:54,834 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 70eda440525723289f2daf7f6251db0d in 460ms, sequenceid=69, compaction requested=false 2023-04-23 05:55:54,834 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:54,835 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:54,836 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@418b5a0b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:54,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:54,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,847 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:54,847 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:54,847 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:54,848 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,848 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:54,849 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,849 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:54,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:54,852 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:54,852 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:54,852 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:54,852 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:54,852 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:54,854 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,854 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,854 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:54,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:54,856 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,857 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:54,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:54,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:54,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:54,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:54,866 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,867 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,867 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,867 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:54,867 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,866 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:54,867 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:54,867 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:54,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:54,967 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:54,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:54,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:54,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:54,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:54,971 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:54,971 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:54,971 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:54,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:54,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,973 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,973 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:54,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:54,973 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:54,973 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:54,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:54,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:54,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:54,974 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:54,975 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:54,975 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:54,975 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:54,975 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:54,975 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:54,975 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:54,975 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-23 05:55:54,975 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:54,975 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:54,976 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:54,976 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:54,976 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:54,976 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:54,986 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/6386884ee76a42e5809a9c93fcf5ad7d 2023-04-23 05:55:55,001 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/f00d095ce37543e3a959f7e9b24349da 2023-04-23 05:55:55,015 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/3f18bf9f6a38412b8e740688deabf2d9 2023-04-23 05:55:55,019 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/6386884ee76a42e5809a9c93fcf5ad7d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/6386884ee76a42e5809a9c93fcf5ad7d 2023-04-23 05:55:55,024 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/6386884ee76a42e5809a9c93fcf5ad7d, entries=150, sequenceid=84, filesize=11.6 K 2023-04-23 05:55:55,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/f00d095ce37543e3a959f7e9b24349da as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f00d095ce37543e3a959f7e9b24349da 2023-04-23 05:55:55,030 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f00d095ce37543e3a959f7e9b24349da, entries=150, sequenceid=84, filesize=11.6 K 2023-04-23 05:55:55,031 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/3f18bf9f6a38412b8e740688deabf2d9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/3f18bf9f6a38412b8e740688deabf2d9 2023-04-23 05:55:55,039 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/3f18bf9f6a38412b8e740688deabf2d9, entries=150, sequenceid=84, filesize=11.6 K 2023-04-23 05:55:55,041 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 70eda440525723289f2daf7f6251db0d in 66ms, sequenceid=84, compaction requested=true 2023-04-23 05:55:55,042 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:55,042 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:55,043 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,044 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,044 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,044 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:55,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a34c396[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:55,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:55,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,047 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,047 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:55,048 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:55,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,051 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:55,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:55,054 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:55,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:55,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:55,054 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:55,054 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:55,055 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,055 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,055 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,056 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,057 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:55,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:55,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,061 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,061 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:55,061 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:55,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:55,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:55,090 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:55:55,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:55,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:55,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:55,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:55,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:55,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:55,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/33e3edc5cc5d4b5c8aebaa304f0950f5 2023-04-23 05:55:55,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229355099, max timestamp=1682229355126Num uniques 859; 2023-04-23 05:55:55,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:55:55,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229355099, max timestamp=1682229355126Num uniques 859; 2023-04-23 05:55:55,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229355099, max timestamp=1682229355126Num uniques 859; 2023-04-23 05:55:55,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:55:55,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:55:55,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229415148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229415152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,161 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:55,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:55,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:55,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:55,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:55,165 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:55,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:55,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:55,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229415153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229415154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229415155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,168 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,168 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:55,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:55,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:55,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:55,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:55,172 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,172 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:55,172 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:55,172 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:55,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,175 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53027a8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:55,176 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:55,176 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,178 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,178 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:55,178 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:55,178 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:55,178 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,178 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:55,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:55,179 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:55,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:55,182 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:55,182 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:55,182 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:55,182 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:55,182 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:55,183 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,183 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,183 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,189 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,189 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,190 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:55,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:55,191 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:55,191 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:55,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229415256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229415262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229415269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229415269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229415269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,291 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:55,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:55,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:55,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:55,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:55,294 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:55,294 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,294 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:55,294 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:55,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,296 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,296 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,296 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:55,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:55,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:55,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:55,301 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,301 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:55,301 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:55,301 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:55,302 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,303 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23a093c3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:55,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:55,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,305 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:55,305 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:55,305 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:55,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:55,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:55,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,307 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:55,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:55,311 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:55,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:55,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:55,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:55,311 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:55,313 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,313 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,313 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,313 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,316 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:55,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:55,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,316 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,317 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,317 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,317 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,317 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:55,317 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,317 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:55,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:55,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,417 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:55,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:55,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:55,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:55,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:55,421 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:55,422 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,422 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:55,422 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:55,422 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,422 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,427 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,428 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,428 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:55,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:55,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:55,429 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:55,430 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,430 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:55,430 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:55,430 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,430 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:55,430 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:55,430 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,430 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:55,430 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:55,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,432 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:55,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3860debe[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:55,434 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:55,434 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,435 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:55,435 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,435 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:55,436 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:55,439 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:55,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:55,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:55,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:55,439 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:55,442 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,442 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,442 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,443 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,445 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:55,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,447 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:55,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,447 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,448 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:55,448 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:55,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229415463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229415467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229415476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229415476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229415477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/1ac624a7d69a4f25b47f4d8271bc6f23 2023-04-23 05:55:55,548 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:55,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:55,551 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:55,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:55,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:55,552 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,552 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,553 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,553 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,553 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,553 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:55,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:55,555 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,555 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:55,555 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:55,555 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:55,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:55,556 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,557 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,557 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:55,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24d48257[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:55,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:55,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,561 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:55,562 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,562 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:55,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:55,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:55,566 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:55,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:55,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:55,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:55,566 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:55,567 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,571 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:55,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:55,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,574 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,574 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,574 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,574 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,574 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:55,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:55,574 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,574 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:55,674 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:55,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:55,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:55,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:55,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:55,677 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:55,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:55,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:55,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,680 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,680 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:55,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:55,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:55,682 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,682 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:55,682 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:55,682 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:55,684 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,684 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@205dfc02[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:55,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:55,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,688 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:55,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:55,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:55,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,689 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:55,692 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:55,692 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:55,692 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:55,692 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:55,692 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:55,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,696 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,699 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,701 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:55,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,701 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:55,701 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:55,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229415769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229415773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229415782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229415783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:55,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229415784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,801 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:55,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:55,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:55,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:55,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:55,805 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:55,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:55,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:55,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,807 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:55,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:55,808 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,808 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:55,808 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:55,808 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:55,813 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,813 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,813 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,813 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:55,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:55,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c44d41[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:55,815 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:55,815 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,816 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,816 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:55,816 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:55,816 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:55,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,817 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:55,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:55,819 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:55,819 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:55,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:55,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:55,820 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:55,822 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,822 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,822 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,824 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,825 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,831 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,831 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,831 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,831 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:55,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:55,831 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,831 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,831 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,831 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:55,831 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,831 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,831 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,831 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/9fdbc0ae304c4e0a9e9a866e4d6e7c9c 2023-04-23 05:55:55,931 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:55,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/33e3edc5cc5d4b5c8aebaa304f0950f5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/33e3edc5cc5d4b5c8aebaa304f0950f5 2023-04-23 05:55:55,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:55,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/33e3edc5cc5d4b5c8aebaa304f0950f5, entries=300, sequenceid=94, filesize=18.5 K 2023-04-23 05:55:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:55,941 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:55,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:55,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:55,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,943 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/1ac624a7d69a4f25b47f4d8271bc6f23 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1ac624a7d69a4f25b47f4d8271bc6f23 2023-04-23 05:55:55,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,943 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,944 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,944 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:55,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:55,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:55,945 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,945 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:55,945 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:55,945 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:55,946 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,947 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1ac624a7d69a4f25b47f4d8271bc6f23, entries=100, sequenceid=94, filesize=9.3 K 2023-04-23 05:55:55,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/9fdbc0ae304c4e0a9e9a866e4d6e7c9c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9fdbc0ae304c4e0a9e9a866e4d6e7c9c 2023-04-23 05:55:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:55,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@396fe629[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:55,957 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:55,957 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,958 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:55,959 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,959 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:55,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9fdbc0ae304c4e0a9e9a866e4d6e7c9c, entries=100, sequenceid=94, filesize=9.3 K 2023-04-23 05:55:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,961 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,962 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 70eda440525723289f2daf7f6251db0d in 872ms, sequenceid=94, compaction requested=true 2023-04-23 05:55:55,962 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,962 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:55,962 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:55,962 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:55,962 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:55,962 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,964 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54663 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:55,964 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:55:55,964 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:55,964 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:55:55,964 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,964 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1d4f0f83ae484153a4fbdf85320be02b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/5a9fd7779a264a28b232c1c0b183d614, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f00d095ce37543e3a959f7e9b24349da, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1ac624a7d69a4f25b47f4d8271bc6f23] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=44.2 K 2023-04-23 05:55:55,964 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:55,964 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a7cc7d2a72974a1d92533eb1e25d9e10, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a80c45db56c04da8a4497f603536427a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/6386884ee76a42e5809a9c93fcf5ad7d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/33e3edc5cc5d4b5c8aebaa304f0950f5] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=53.4 K 2023-04-23 05:55:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:55,965 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1d4f0f83ae484153a4fbdf85320be02b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229352666 2023-04-23 05:55:55,965 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:55,965 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:55,965 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:55,965 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:55,965 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:55,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:55,965 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a7cc7d2a72974a1d92533eb1e25d9e10, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229352666 2023-04-23 05:55:55,965 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 5a9fd7779a264a28b232c1c0b183d614, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682229353818 2023-04-23 05:55:55,965 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a80c45db56c04da8a4497f603536427a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682229353818 2023-04-23 05:55:55,966 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 6386884ee76a42e5809a9c93fcf5ad7d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682229354468 2023-04-23 05:55:55,966 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f00d095ce37543e3a959f7e9b24349da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682229354468 2023-04-23 05:55:55,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:55,966 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,967 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1ac624a7d69a4f25b47f4d8271bc6f23, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1682229355088 2023-04-23 05:55:55,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,967 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,967 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 33e3edc5cc5d4b5c8aebaa304f0950f5, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1682229355084 2023-04-23 05:55:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:55,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:55,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:55,971 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,972 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:55,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:55,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:55,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:55,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:55,975 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:55,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:55,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:55,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:55,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:55,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:55,977 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:55,977 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:55,978 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:55,991 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#410 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:56,002 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/63ebccf4ad9d41369a946b9a967d93b9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/63ebccf4ad9d41369a946b9a967d93b9 2023-04-23 05:55:56,006 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into 63ebccf4ad9d41369a946b9a967d93b9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:56,006 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:56,006 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=12, startTime=1682229355962; duration=0sec 2023-04-23 05:55:56,006 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:56,006 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:55:56,007 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45303 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:55:56,008 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/C is initiating minor compaction (all files) 2023-04-23 05:55:56,008 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/C in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:56,008 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/e34cbbba0ff7498da8086e175184416c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c59f418dadf466a822241390037cc24, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/3f18bf9f6a38412b8e740688deabf2d9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9fdbc0ae304c4e0a9e9a866e4d6e7c9c] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=44.2 K 2023-04-23 05:55:56,008 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e34cbbba0ff7498da8086e175184416c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1682229352666 2023-04-23 05:55:56,008 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1c59f418dadf466a822241390037cc24, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1682229353818 2023-04-23 05:55:56,009 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3f18bf9f6a38412b8e740688deabf2d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682229354468 2023-04-23 05:55:56,009 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9fdbc0ae304c4e0a9e9a866e4d6e7c9c, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1682229355088 2023-04-23 05:55:56,014 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#C#compaction#411 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:56,077 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:56,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:56,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:56,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:56,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:56,081 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:56,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:56,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:56,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:56,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:56,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:56,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:56,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:56,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:56,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:56,084 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:56,085 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:56,085 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:56,085 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:56,085 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-23 05:55:56,085 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:56,085 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:56,085 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:56,085 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:56,085 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:56,085 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:56,094 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/a1879bec24d6483cb9a4e8dbd8f7eacf 2023-04-23 05:55:56,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:56,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:56,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229416292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229416294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229416296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229416299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229416300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,397 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/f58e8da79bc141839f80a04d5a5f5c40 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f58e8da79bc141839f80a04d5a5f5c40 2023-04-23 05:55:56,402 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into f58e8da79bc141839f80a04d5a5f5c40(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:56,402 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:56,402 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=12, startTime=1682229355962; duration=0sec 2023-04-23 05:55:56,402 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:56,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229416401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229416404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229416405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229416410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,426 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/915805e477824db58d147aff06b1afd0 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/915805e477824db58d147aff06b1afd0 2023-04-23 05:55:56,430 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70eda440525723289f2daf7f6251db0d/C of 70eda440525723289f2daf7f6251db0d into 915805e477824db58d147aff06b1afd0(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:56,430 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:56,430 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/C, priority=12, startTime=1682229355962; duration=0sec 2023-04-23 05:55:56,430 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:56,512 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/7a719c0f74e8493abec430ec2bbf557b 2023-04-23 05:55:56,522 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/97dd85be66cd4e9090d1757a75f94851 2023-04-23 05:55:56,526 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/a1879bec24d6483cb9a4e8dbd8f7eacf as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a1879bec24d6483cb9a4e8dbd8f7eacf 2023-04-23 05:55:56,529 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a1879bec24d6483cb9a4e8dbd8f7eacf, entries=150, sequenceid=130, filesize=11.6 K 2023-04-23 05:55:56,530 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/7a719c0f74e8493abec430ec2bbf557b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7a719c0f74e8493abec430ec2bbf557b 2023-04-23 05:55:56,537 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7a719c0f74e8493abec430ec2bbf557b, entries=150, sequenceid=130, filesize=11.6 K 2023-04-23 05:55:56,538 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/97dd85be66cd4e9090d1757a75f94851 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/97dd85be66cd4e9090d1757a75f94851 2023-04-23 05:55:56,541 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/97dd85be66cd4e9090d1757a75f94851, entries=150, sequenceid=130, filesize=11.6 K 2023-04-23 05:55:56,542 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 70eda440525723289f2daf7f6251db0d in 457ms, sequenceid=130, compaction requested=false 2023-04-23 05:55:56,542 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:56,542 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:56,549 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,549 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,549 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,549 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:56,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39a90d69[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:56,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:56,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,552 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:56,553 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,554 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:56,554 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,554 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:56,554 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,554 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:56,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:56,558 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:56,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:56,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:56,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:56,558 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:56,559 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,559 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,559 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,563 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,565 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,565 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,565 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:56,565 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:56,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:56,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:56,566 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,566 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,566 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,566 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,566 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:56,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:56,566 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:56,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:56,566 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:56,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:56,611 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:55:56,612 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:56,612 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:56,612 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:56,612 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:56,612 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:56,612 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:56,624 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/38dffb138eef4dbd8d3e40194f9236ae 2023-04-23 05:55:56,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229416644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229356641, max timestamp=1682229356652Num uniques 859; 2023-04-23 05:55:56,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229356641, max timestamp=1682229356652Num uniques 859; 2023-04-23 05:55:56,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:55:56,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:55:56,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229356641, max timestamp=1682229356652Num uniques 859; 2023-04-23 05:55:56,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:55:56,666 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:56,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:56,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:56,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:56,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:56,670 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:56,670 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:56,670 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:56,670 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:56,671 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,671 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,672 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:56,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:56,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:56,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:56,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:56,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:56,674 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:56,674 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:56,674 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:56,674 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:56,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76db4bb2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:56,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:56,678 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,679 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:56,680 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,680 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:56,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:56,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:56,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:56,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:56,688 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:56,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:56,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:56,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:56,688 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:56,689 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,689 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,690 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:56,690 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,691 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,693 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:56,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:56,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:56,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:56,694 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:56,694 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:56,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:56,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:56,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:56,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:56,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229416690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229416691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229416700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229416755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,794 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:56,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:56,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:56,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:56,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:56,797 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:56,797 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:56,797 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:56,797 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:56,798 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,798 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,799 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,799 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:56,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,799 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,799 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:56,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:56,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:56,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:56,800 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:56,801 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:56,801 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:56,801 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:56,801 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:56,801 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:56,801 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:56,801 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,801 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:56,801 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:56,804 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,804 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:56,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53d0ae6d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:56,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:56,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,807 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,807 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:56,807 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:56,807 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:56,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229416801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:56,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229416801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:56,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:56,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:56,811 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:56,811 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:56,811 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:56,811 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:56,811 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:56,813 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,813 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,813 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:56,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:56,813 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229416809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:56,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:56,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,819 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:56,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:56,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:56,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:56,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:56,820 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:56,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,820 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:56,920 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:56,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:56,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:56,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:56,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:56,924 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:56,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:56,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:56,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:56,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,925 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:56,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:56,926 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,926 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:56,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:56,927 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:56,927 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:56,927 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:56,927 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:56,927 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:56,928 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:56,928 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:56,928 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:56,928 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:56,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:56,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:56,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:56,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:56,929 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,930 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,930 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:56,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15484490[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:56,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:56,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,932 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,932 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:56,932 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,932 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:56,932 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:56,932 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:56,935 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:56,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:56,939 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:56,939 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:56,939 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:56,939 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:56,939 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:56,940 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,940 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:56,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,940 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:56,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:56,941 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,942 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:56,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:56,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:56,945 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:56,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:56,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:56,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:56,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:56,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,946 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:56,946 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:56,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:56,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:56,946 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:56,946 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:56,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:56,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229416959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229417009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229417010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229417018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,046 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:57,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:57,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:57,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:57,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:57,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/831494818ea04158a00b67b4b72c8513 2023-04-23 05:55:57,049 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:57,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:57,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:57,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,050 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:57,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:57,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:57,053 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,053 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:57,054 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:57,054 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:57,057 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,058 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:57,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647bc115[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:57,059 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:57,059 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,060 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:57,060 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:57,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:57,061 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:57,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:57,064 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:57,064 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:57,064 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:57,064 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:57,064 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:57,070 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,070 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,070 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,071 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,072 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,074 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,074 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,074 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,074 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,075 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:57,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:57,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:57,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:57,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:57,075 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:57,075 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:57,075 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:57,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:57,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:57,175 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:57,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:57,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:57,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:57,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:57,179 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:57,179 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,179 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:57,179 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:57,179 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,179 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:57,180 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:57,181 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,181 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:57,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:57,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:57,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:57,182 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:57,182 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,182 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:57,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:57,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:57,183 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,183 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:57,183 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:57,183 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:57,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:57,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:57,183 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:57,184 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,184 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,185 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:57,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:57,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@485180f3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:57,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,190 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:57,190 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,190 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:57,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:57,191 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:57,194 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:57,194 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:57,194 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:57,194 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:57,194 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:57,195 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,195 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,195 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:57,196 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,197 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:57,200 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:57,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:57,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:57,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:57,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:57,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,200 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:57,200 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:57,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,201 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:57,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229417267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,301 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:57,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:57,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:57,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:57,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:57,304 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:57,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:57,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:57,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,306 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,307 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:57,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:57,307 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,307 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:57,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:57,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:57,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:57,308 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,309 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:57,309 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:57,309 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,309 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:57,309 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:57,309 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,309 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:57,309 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:57,310 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,311 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,311 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38344 deadline: 1682229417306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:57,313 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:57,313 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ea4ec52[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:57,314 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:57,314 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:57,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,314 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:57,315 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:57,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:57,318 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:57,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:57,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:57,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:57,318 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:57,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229417316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229417317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,321 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,321 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,321 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:57,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:57,321 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,323 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,325 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:57,325 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:57,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:57,325 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:57,325 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:57,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:57,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:57,325 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,325 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:57,325 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:57,325 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,325 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,325 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,325 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,325 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229417326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,425 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:57,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:57,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:57,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:57,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:57,429 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:57,429 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,429 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:57,429 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:57,429 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,429 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,431 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:57,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:57,431 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,431 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,431 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:57,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:57,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:57,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:57,432 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:57,433 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,433 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:57,433 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:57,433 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:57,434 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:57,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@499f47d2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:57,436 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:57,436 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,438 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:57,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:57,439 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:57,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:57,442 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:57,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:57,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:57,442 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:57,442 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:57,443 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,443 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:57,443 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:57,444 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:57,445 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:57,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:57,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:57,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:57,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:57,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:57,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:57,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:57,451 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:57,452 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:57,452 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:57,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:57,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/84d6f0dcea084cfb9b7364ce679cacee 2023-04-23 05:55:57,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/38dffb138eef4dbd8d3e40194f9236ae as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/38dffb138eef4dbd8d3e40194f9236ae 2023-04-23 05:55:57,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/38dffb138eef4dbd8d3e40194f9236ae, entries=200, sequenceid=143, filesize=14.1 K 2023-04-23 05:55:57,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/831494818ea04158a00b67b4b72c8513 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/831494818ea04158a00b67b4b72c8513 2023-04-23 05:55:57,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/831494818ea04158a00b67b4b72c8513, entries=150, sequenceid=143, filesize=11.8 K 2023-04-23 05:55:57,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/84d6f0dcea084cfb9b7364ce679cacee as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/84d6f0dcea084cfb9b7364ce679cacee 2023-04-23 05:55:57,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/84d6f0dcea084cfb9b7364ce679cacee, entries=150, sequenceid=143, filesize=11.8 K 2023-04-23 05:55:57,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 70eda440525723289f2daf7f6251db0d in 879ms, sequenceid=143, compaction requested=true 2023-04-23 05:55:57,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:57,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:57,490 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:57,490 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:57,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:57,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:57,492 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38438 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:57,492 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:55:57,492 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,492 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f58e8da79bc141839f80a04d5a5f5c40, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a1879bec24d6483cb9a4e8dbd8f7eacf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/38dffb138eef4dbd8d3e40194f9236ae] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=37.5 K 2023-04-23 05:55:57,493 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:57,493 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:55:57,493 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,493 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/63ebccf4ad9d41369a946b9a967d93b9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7a719c0f74e8493abec430ec2bbf557b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/831494818ea04158a00b67b4b72c8513] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=35.2 K 2023-04-23 05:55:57,493 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting f58e8da79bc141839f80a04d5a5f5c40, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1682229354468 2023-04-23 05:55:57,493 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 63ebccf4ad9d41369a946b9a967d93b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1682229354468 2023-04-23 05:55:57,494 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting a1879bec24d6483cb9a4e8dbd8f7eacf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682229355149 2023-04-23 05:55:57,494 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7a719c0f74e8493abec430ec2bbf557b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682229355149 2023-04-23 05:55:57,494 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 38dffb138eef4dbd8d3e40194f9236ae, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682229356282 2023-04-23 05:55:57,494 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 831494818ea04158a00b67b4b72c8513, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682229356295 2023-04-23 05:55:57,502 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:57,502 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#418 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:57,532 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/9a3a528ae72f4dd2947cd151dabbea89 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9a3a528ae72f4dd2947cd151dabbea89 2023-04-23 05:55:57,540 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into 9a3a528ae72f4dd2947cd151dabbea89(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:57,540 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/4a4c31dc70c14e9ebc9cf9eb2d199560 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/4a4c31dc70c14e9ebc9cf9eb2d199560 2023-04-23 05:55:57,540 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:57,540 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=13, startTime=1682229357490; duration=0sec 2023-04-23 05:55:57,540 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:57,541 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:57,542 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36048 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:57,544 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/C is initiating minor compaction (all files) 2023-04-23 05:55:57,544 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/C in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,544 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/915805e477824db58d147aff06b1afd0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/97dd85be66cd4e9090d1757a75f94851, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/84d6f0dcea084cfb9b7364ce679cacee] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=35.2 K 2023-04-23 05:55:57,545 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 915805e477824db58d147aff06b1afd0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1682229354468 2023-04-23 05:55:57,545 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into 4a4c31dc70c14e9ebc9cf9eb2d199560(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:57,545 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:57,545 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=13, startTime=1682229357490; duration=0sec 2023-04-23 05:55:57,545 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:57,545 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 97dd85be66cd4e9090d1757a75f94851, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1682229355149 2023-04-23 05:55:57,546 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 84d6f0dcea084cfb9b7364ce679cacee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682229356295 2023-04-23 05:55:57,551 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#C#compaction#420 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:57,552 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:57,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:57,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:57,556 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:57,556 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,556 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:57,556 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:57,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,558 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:57,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:57,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:57,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:57,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:57,560 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:57,560 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:57,563 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-23 05:55:57,564 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:57,564 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:57,564 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:57,564 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/5d1ea8c6398c454d9feb2d0d610ff4db as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5d1ea8c6398c454d9feb2d0d610ff4db 2023-04-23 05:55:57,564 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:57,564 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:57,564 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:57,571 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/C of 70eda440525723289f2daf7f6251db0d into 5d1ea8c6398c454d9feb2d0d610ff4db(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:57,571 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:57,572 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/C, priority=13, startTime=1682229357490; duration=0sec 2023-04-23 05:55:57,572 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:57,595 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/0a73a6c0b5574a7d840f1571e51f78ff 2023-04-23 05:55:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:57,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:57,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229417824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229417824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229417825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229417837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:57,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:57,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229417931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,006 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/be06530a39424970aebf66789ce18ab8 2023-04-23 05:55:58,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:58,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229418136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,416 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=179 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/7f70ecea01f04cc6abcf53a2f47429fa 2023-04-23 05:55:58,420 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/0a73a6c0b5574a7d840f1571e51f78ff as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/0a73a6c0b5574a7d840f1571e51f78ff 2023-04-23 05:55:58,424 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/0a73a6c0b5574a7d840f1571e51f78ff, entries=150, sequenceid=179, filesize=11.8 K 2023-04-23 05:55:58,425 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/be06530a39424970aebf66789ce18ab8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/be06530a39424970aebf66789ce18ab8 2023-04-23 05:55:58,428 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/be06530a39424970aebf66789ce18ab8, entries=150, sequenceid=179, filesize=11.8 K 2023-04-23 05:55:58,429 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/7f70ecea01f04cc6abcf53a2f47429fa as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/7f70ecea01f04cc6abcf53a2f47429fa 2023-04-23 05:55:58,432 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/7f70ecea01f04cc6abcf53a2f47429fa, entries=150, sequenceid=179, filesize=11.8 K 2023-04-23 05:55:58,433 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 70eda440525723289f2daf7f6251db0d in 870ms, sequenceid=179, compaction requested=false 2023-04-23 05:55:58,433 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:58,433 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:58,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:58,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:58,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:58,433 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:58,435 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73d6549d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:58,437 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:58,437 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,438 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:58,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:58,438 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:58,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:58,441 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:58,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:58,443 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:58,443 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:58,444 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:58,444 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:58,444 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:58,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,445 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,446 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:55:58,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:58,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:58,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:58,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:58,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:55:58,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,447 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:58,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,452 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,452 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,452 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,452 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,452 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,452 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:58,452 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,453 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:58,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:58,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:58,453 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:58,453 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:58,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/cf9c2e5bfcd4402590184be122e9db8f 2023-04-23 05:55:58,553 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:58,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:58,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:58,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:58,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:58,556 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,557 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,558 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:58,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:58,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:58,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:58,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:58,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:58,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:58,559 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:58,560 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:58,560 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:58,560 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:58,560 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:58,560 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:58,563 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,563 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:58,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:58,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229418555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:58,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3744140c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:58,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:58,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,567 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:58,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:58,569 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,569 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:58,569 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,569 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:58,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:58,571 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:58,571 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:58,571 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:58,571 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:58,571 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:58,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,573 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,573 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,575 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,577 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:58,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:58,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:58,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:58,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,578 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:58,578 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:58,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,578 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,671 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:58,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229418666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,678 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:58,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:58,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:58,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:58,681 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:58,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:58,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:58,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:58,682 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,682 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,683 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:58,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:58,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,683 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:58,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:58,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:58,684 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:58,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:58,685 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:58,685 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:58,685 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:58,685 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:58,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:58,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ebbb8ca[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:58,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:58,689 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,689 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:58,690 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:58,690 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,690 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:58,692 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:58,692 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,692 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,692 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:58,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:58,694 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:58,694 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:58,694 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:58,694 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:58,694 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:58,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,697 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,699 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:58,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:58,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:58,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:58,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:58,701 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,701 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:58,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,701 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:58,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,702 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,802 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:58,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:58,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:58,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:58,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:58,805 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:58,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:58,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:58,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:58,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,809 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:58,809 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:58,810 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,810 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:58,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:58,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:58,811 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:58,811 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:58,811 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:58,811 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,814 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:58,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da9df18[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:58,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:58,817 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,817 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:58,818 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:58,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:58,820 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:58,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:58,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:58,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:58,820 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:58,824 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,825 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,826 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,827 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,827 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,830 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:58,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:58,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:58,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:58,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:58,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,830 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,831 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:58,831 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:58,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:58,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229418836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:58,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229418837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:58,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229418848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/68232384752649468a74816c9e668102 2023-04-23 05:55:58,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:58,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229418872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,931 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:58,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:58,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:58,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:58,935 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:58,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:58,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:58,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:58,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,936 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:58,936 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,937 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,937 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:58,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:58,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:58,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:58,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:58,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:58,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:58,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:58,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:58,939 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:58,939 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:58,939 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:58,939 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:58,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:58,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:58,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:58,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:58,941 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,941 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,941 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,941 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:58,942 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:58,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1a168c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:58,942 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,943 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:58,944 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,944 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:58,946 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:58,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:58,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:58,948 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:58,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:58,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:58,948 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:58,948 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:58,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:58,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:58,950 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,952 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,953 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:58,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:58,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,954 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:58,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:58,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:58,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:58,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:58,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:58,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:58,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,954 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:58,954 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:58,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:58,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:58,954 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,054 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:59,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:59,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:59,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:59,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:59,058 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:59,058 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:59,058 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:59,058 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:59,059 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,059 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,060 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:59,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:59,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:59,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:59,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:59,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:59,061 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:59,061 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:59,062 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:59,062 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:59,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:59,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:59,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:59,062 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:59,064 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:59,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:59,066 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:59,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b699977[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:59,066 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,067 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,067 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:59,067 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:59,067 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:59,067 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,067 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:59,068 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:59,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:59,070 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:59,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:59,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:59,070 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:59,070 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:59,072 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,072 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:59,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:59,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,072 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:59,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:59,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:59,073 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,074 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:59,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:59,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:59,078 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:59,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:59,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:59,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:59,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,078 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:59,078 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:59,078 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,179 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:59,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:59,182 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:59,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:59,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:59,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:59,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:55:59,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,183 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38368 deadline: 1682229419180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,184 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:59,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:59,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:59,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:59,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:59,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:59,186 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:59,188 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:59,188 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:59,188 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:59,188 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:55:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:55:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:55:59,188 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:55:59,192 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,193 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:59,193 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,193 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:59,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:55:59,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38470138[Count = 0] remaining members to acquire global barrier 2023-04-23 05:55:59,194 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:55:59,194 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,202 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,202 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:55:59,202 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:55:59,202 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,202 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:55:59,202 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:55:59,203 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:55:59,203 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,203 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:59,204 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:55:59,206 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:55:59,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:55:59,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:55:59,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:55:59,206 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:55:59,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:59,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,208 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:55:59,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:55:59,209 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,210 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:59,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:55:59,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:59,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:55:59,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:55:59,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:55:59,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:55:59,212 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:55:59,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:55:59,212 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:55:59,213 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:55:59,277 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/b2449c246209427e9f01cef66722a8a2 2023-04-23 05:55:59,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/cf9c2e5bfcd4402590184be122e9db8f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cf9c2e5bfcd4402590184be122e9db8f 2023-04-23 05:55:59,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cf9c2e5bfcd4402590184be122e9db8f, entries=200, sequenceid=192, filesize=14.1 K 2023-04-23 05:55:59,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/68232384752649468a74816c9e668102 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/68232384752649468a74816c9e668102 2023-04-23 05:55:59,288 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/68232384752649468a74816c9e668102, entries=150, sequenceid=192, filesize=11.8 K 2023-04-23 05:55:59,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/b2449c246209427e9f01cef66722a8a2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b2449c246209427e9f01cef66722a8a2 2023-04-23 05:55:59,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b2449c246209427e9f01cef66722a8a2, entries=150, sequenceid=192, filesize=11.8 K 2023-04-23 05:55:59,294 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 70eda440525723289f2daf7f6251db0d in 848ms, sequenceid=192, compaction requested=true 2023-04-23 05:55:59,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:59,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:59,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:59,294 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:59,294 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:59,294 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-23 05:55:59,295 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:59,295 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:59,295 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:55:59,295 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:55:59,295 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:59,295 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:59,296 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9a3a528ae72f4dd2947cd151dabbea89, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/0a73a6c0b5574a7d840f1571e51f78ff, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cf9c2e5bfcd4402590184be122e9db8f] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=37.9 K 2023-04-23 05:55:59,296 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/4a4c31dc70c14e9ebc9cf9eb2d199560, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/be06530a39424970aebf66789ce18ab8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/68232384752649468a74816c9e668102] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=35.6 K 2023-04-23 05:55:59,296 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 9a3a528ae72f4dd2947cd151dabbea89, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682229356295 2023-04-23 05:55:59,296 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 4a4c31dc70c14e9ebc9cf9eb2d199560, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682229356295 2023-04-23 05:55:59,296 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 0a73a6c0b5574a7d840f1571e51f78ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682229356678 2023-04-23 05:55:59,296 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting be06530a39424970aebf66789ce18ab8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682229356678 2023-04-23 05:55:59,297 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting cf9c2e5bfcd4402590184be122e9db8f, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682229357803 2023-04-23 05:55:59,297 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 68232384752649468a74816c9e668102, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682229357803 2023-04-23 05:55:59,304 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#427 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:59,306 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#428 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:59,313 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:55:59,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:55:59,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:55:59,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:55:59,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:55:59,316 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:55:59,316 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:59,316 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:55:59,316 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:55:59,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,319 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:55:59,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:55:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:55:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,320 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:55:59,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:55:59,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:55:59,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:55:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:55:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:55:59,321 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:55:59,321 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/9ae92832623148a8bd1f1f3b1116fe48 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9ae92832623148a8bd1f1f3b1116fe48 2023-04-23 05:55:59,321 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:55:59,322 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:55:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:55:59,322 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:55:59,322 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:59,322 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:55:59,322 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=114.05 KB heapSize=339.89 KB 2023-04-23 05:55:59,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229358539, max timestamp=1682229359320Num uniques 859; 2023-04-23 05:55:59,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229358539, max timestamp=1682229359320Num uniques 859; 2023-04-23 05:55:59,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:59,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:59,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:55:59,324 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:55:59,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229358539, max timestamp=1682229359320Num uniques 859; 2023-04-23 05:55:59,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:59,324 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:55:59,324 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:59,324 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:55:59,325 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:59,325 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:55:59,325 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:55:59,327 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into 9ae92832623148a8bd1f1f3b1116fe48(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:59,327 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:59,327 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=13, startTime=1682229359294; duration=0sec 2023-04-23 05:55:59,327 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:55:59,327 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:55:59,328 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/c7a93e6df8d64dc2b7c0bd13007dee14 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c7a93e6df8d64dc2b7c0bd13007dee14 2023-04-23 05:55:59,329 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:55:59,330 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/C is initiating minor compaction (all files) 2023-04-23 05:55:59,330 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/C in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:55:59,330 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5d1ea8c6398c454d9feb2d0d610ff4db, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/7f70ecea01f04cc6abcf53a2f47429fa, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b2449c246209427e9f01cef66722a8a2] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=35.6 K 2023-04-23 05:55:59,331 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 5d1ea8c6398c454d9feb2d0d610ff4db, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1682229356295 2023-04-23 05:55:59,331 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 7f70ecea01f04cc6abcf53a2f47429fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=179, earliestPutTs=1682229356678 2023-04-23 05:55:59,331 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b2449c246209427e9f01cef66722a8a2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682229357803 2023-04-23 05:55:59,335 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into c7a93e6df8d64dc2b7c0bd13007dee14(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:59,335 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:59,335 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=13, startTime=1682229359294; duration=0sec 2023-04-23 05:55:59,335 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:59,341 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#C#compaction#430 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:55:59,343 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/14557c4db40145ada919d4c94050af03 2023-04-23 05:55:59,350 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/2e31d35cb1134ea39d1aa9fdf1e4d620 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2e31d35cb1134ea39d1aa9fdf1e4d620 2023-04-23 05:55:59,354 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/C of 70eda440525723289f2daf7f6251db0d into 2e31d35cb1134ea39d1aa9fdf1e4d620(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:55:59,354 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:55:59,354 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/C, priority=13, startTime=1682229359294; duration=0sec 2023-04-23 05:55:59,354 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:55:59,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229359390, max timestamp=1682229359411Num uniques 859; 2023-04-23 05:55:59,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:59,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229359390, max timestamp=1682229359411Num uniques 859; 2023-04-23 05:55:59,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:59,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229359390, max timestamp=1682229359411Num uniques 859; 2023-04-23 05:55:59,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:55:59,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682229359484, max timestamp=1682229359512Num uniques 1018; 2023-04-23 05:55:59,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-23 05:55:59,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682229359484, max timestamp=1682229359512Num uniques 1018; 2023-04-23 05:55:59,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682229359484, max timestamp=1682229359512Num uniques 1018; 2023-04-23 05:55:59,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-23 05:55:59,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-23 05:55:59,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682229359581, max timestamp=1682229359608Num uniques 1077; 2023-04-23 05:55:59,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682229359581, max timestamp=1682229359608Num uniques 1077; 2023-04-23 05:55:59,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-23 05:55:59,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682229359581, max timestamp=1682229359608Num uniques 1077; 2023-04-23 05:55:59,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-23 05:55:59,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-23 05:55:59,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682229359687, max timestamp=1682229359700Num uniques 1086; 2023-04-23 05:55:59,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-23 05:55:59,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682229359687, max timestamp=1682229359700Num uniques 1086; 2023-04-23 05:55:59,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-23 05:55:59,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1682229359687, max timestamp=1682229359700Num uniques 1086; 2023-04-23 05:55:59,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-23 05:55:59,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682229359738, max timestamp=1682229359759Num uniques 1145; 2023-04-23 05:55:59,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-23 05:55:59,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682229359738, max timestamp=1682229359759Num uniques 1145; 2023-04-23 05:55:59,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-23 05:55:59,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682229359738, max timestamp=1682229359759Num uniques 1145; 2023-04-23 05:55:59,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-23 05:55:59,763 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/b6ce4f6aecbd4806b96e5ee5766ae09f 2023-04-23 05:55:59,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682229359804, max timestamp=1682229359822Num uniques 1054; 2023-04-23 05:55:59,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-23 05:55:59,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682229359804, max timestamp=1682229359822Num uniques 1054; 2023-04-23 05:55:59,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1682229359804, max timestamp=1682229359822Num uniques 1054; 2023-04-23 05:55:59,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-23 05:55:59,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-23 05:55:59,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682229359866, max timestamp=1682229359887Num uniques 1063; 2023-04-23 05:55:59,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682229359866, max timestamp=1682229359887Num uniques 1063; 2023-04-23 05:55:59,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-23 05:55:59,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1682229359866, max timestamp=1682229359887Num uniques 1063; 2023-04-23 05:55:59,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-23 05:55:59,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-23 05:55:59,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682229359939, max timestamp=1682229359953Num uniques 1072; 2023-04-23 05:55:59,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682229359939, max timestamp=1682229359953Num uniques 1072; 2023-04-23 05:55:59,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-23 05:55:59,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-23 05:55:59,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682229359939, max timestamp=1682229359953Num uniques 1072; 2023-04-23 05:55:59,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-23 05:56:00,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682229359988, max timestamp=1682229360001Num uniques 1031; 2023-04-23 05:56:00,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682229359988, max timestamp=1682229360001Num uniques 1031; 2023-04-23 05:56:00,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-23 05:56:00,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1682229359988, max timestamp=1682229360001Num uniques 1031; 2023-04-23 05:56:00,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-23 05:56:00,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-23 05:56:00,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682229360034, max timestamp=1682229360051Num uniques 1090; 2023-04-23 05:56:00,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-23 05:56:00,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682229360034, max timestamp=1682229360051Num uniques 1090; 2023-04-23 05:56:00,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-23 05:56:00,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682229360034, max timestamp=1682229360051Num uniques 1090; 2023-04-23 05:56:00,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-23 05:56:00,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682229360089, max timestamp=1682229360103Num uniques 1049; 2023-04-23 05:56:00,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-23 05:56:00,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682229360089, max timestamp=1682229360103Num uniques 1049; 2023-04-23 05:56:00,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682229360089, max timestamp=1682229360103Num uniques 1049; 2023-04-23 05:56:00,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-23 05:56:00,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-23 05:56:00,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1682229360145, max timestamp=1682229360164Num uniques 1058; 2023-04-23 05:56:00,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1682229360145, max timestamp=1682229360164Num uniques 1058; 2023-04-23 05:56:00,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-23 05:56:00,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1682229360145, max timestamp=1682229360164Num uniques 1058; 2023-04-23 05:56:00,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-23 05:56:00,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-04-23 05:56:00,173 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/d5add1b081e04375868504bd83e55b45 2023-04-23 05:56:00,177 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/14557c4db40145ada919d4c94050af03 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/14557c4db40145ada919d4c94050af03 2023-04-23 05:56:00,181 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/14557c4db40145ada919d4c94050af03, entries=150, sequenceid=213, filesize=11.8 K 2023-04-23 05:56:00,182 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/b6ce4f6aecbd4806b96e5ee5766ae09f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/b6ce4f6aecbd4806b96e5ee5766ae09f 2023-04-23 05:56:00,185 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/b6ce4f6aecbd4806b96e5ee5766ae09f, entries=150, sequenceid=213, filesize=11.8 K 2023-04-23 05:56:00,186 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/d5add1b081e04375868504bd83e55b45 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d5add1b081e04375868504bd83e55b45 2023-04-23 05:56:00,189 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d5add1b081e04375868504bd83e55b45, entries=150, sequenceid=213, filesize=11.8 K 2023-04-23 05:56:00,190 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=80.51 KB/82440 for 70eda440525723289f2daf7f6251db0d in 868ms, sequenceid=213, compaction requested=false 2023-04-23 05:56:00,190 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:00,190 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:00,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:00,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:00,190 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,191 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b654837[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:00,193 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:00,193 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,194 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:00,194 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,195 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:00,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:00,195 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=93.93 KB heapSize=256.17 KB 2023-04-23 05:56:00,195 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:00,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:00,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:00,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:00,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:00,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:00,196 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:00,199 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:00,199 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:00,199 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:00,199 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:00,199 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:00,201 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,201 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,201 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,202 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,204 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/5938e222fa314da382d3c3f96ee992aa 2023-04-23 05:56:00,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,206 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:00,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:00,206 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,207 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:00,207 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:00,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,207 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229420247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229420252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,307 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:00,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:00,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:00,310 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:00,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:00,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:00,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,312 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,312 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,313 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,313 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:00,315 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,315 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:00,315 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:00,315 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,315 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:00,316 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:00,317 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:00,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67e799a1[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:00,319 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:00,319 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,321 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:00,322 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,322 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:00,323 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:00,323 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,323 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,323 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:00,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:00,326 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:00,326 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:00,326 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:00,326 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:00,326 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:00,327 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,327 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,328 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,328 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,329 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,331 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:00,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:00,331 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,332 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,332 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,332 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,332 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,332 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:00,332 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:00,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:00,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229420355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229420360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,432 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:00,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:00,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:00,436 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:00,436 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,436 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:00,436 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:00,437 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,437 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,439 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,439 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:00,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:00,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:00,441 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,441 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:00,441 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:00,441 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:00,442 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,442 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,443 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,443 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4062160b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:00,444 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:00,444 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,446 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:00,446 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,446 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:00,451 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,451 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:00,452 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,452 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:00,454 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:00,454 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:00,454 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:00,454 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:00,454 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:00,456 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,456 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,456 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,457 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,459 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,461 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,461 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,461 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,461 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,461 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:00,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:00,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:00,461 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,461 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,462 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,462 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,462 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,462 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:00,462 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:00,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,562 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:00,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:00,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229420559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:00,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:00,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:00,565 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:00,565 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,565 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:00,565 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:00,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,567 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,567 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,567 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:00,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:00,569 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:00,569 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,570 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:00,570 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:00,570 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,570 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:00,570 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:00,570 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,570 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:00,570 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:00,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229420568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,572 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,573 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:00,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@402c91fe[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:00,576 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:00,576 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,577 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:00,577 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,578 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:00,578 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,579 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:00,583 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:00,583 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:00,583 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:00,583 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:00,583 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:00,584 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,584 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,584 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,585 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,587 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,592 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:00,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:00,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:00,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,592 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:00,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,592 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:00,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,622 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/1879b28da4d840bf84695e6027c35473 2023-04-23 05:56:00,693 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:00,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:00,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:00,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:00,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:00,696 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:00,696 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,696 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:00,696 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:00,697 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,697 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:00,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:00,699 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:00,699 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,699 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:00,700 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,700 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:00,700 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:00,700 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:00,700 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:00,701 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,701 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,702 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,702 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:00,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21694d3c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:00,704 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:00,704 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,705 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,705 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:00,705 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:00,705 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:00,706 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,706 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:00,708 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,708 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:00,708 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,708 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:00,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:00,712 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:00,712 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:00,712 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:00,712 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:00,712 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:00,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,714 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,715 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,718 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,718 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,719 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:00,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:00,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:00,719 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:00,720 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:00,820 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:00,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:00,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:00,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:00,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:00,823 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:00,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:00,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:00,824 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,824 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,825 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,825 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,826 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,826 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:00,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:00,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:00,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:00,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:00,826 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:00,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:00,827 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,827 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:00,827 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:00,827 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:00,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:00,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:00,827 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:00,829 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,829 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,829 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,829 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:00,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:00,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a18f221[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:00,831 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:00,831 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,832 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:00,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:00,832 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:00,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,832 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,833 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,833 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:00,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:00,836 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:00,836 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:00,836 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:00,836 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:00,836 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:00,840 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,840 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,841 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,842 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,844 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:00,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,847 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:00,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,847 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:00,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229420845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,852 DEBUG [Thread-3031] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:00,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229420857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,862 DEBUG [Thread-3033] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:00,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229420866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229420866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,872 DEBUG [Thread-3035] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:00,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:00,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229420875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,947 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:00,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:00,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:00,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:00,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:00,951 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:00,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:00,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:00,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,951 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,952 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,952 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:00,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:00,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:00,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:00,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:00,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:00,955 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:00,955 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:00,957 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,957 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,957 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,957 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:00,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a9a6127[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:00,959 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:00,959 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,961 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,961 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:00,962 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:00,962 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,962 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:00,962 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:00,963 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,963 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:00,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:00,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:00,967 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:00,967 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:00,967 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:00,967 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:00,967 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:00,969 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,969 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,969 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:00,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:00,969 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,971 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,973 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:00,973 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:00,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:00,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:00,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:00,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:00,973 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:00,973 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:00,973 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,974 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:00,974 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:00,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:00,974 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:00,974 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,974 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:00,974 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:00,974 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:01,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1f4745474d724f7886382245dcced426 2023-04-23 05:56:01,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/5938e222fa314da382d3c3f96ee992aa as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5938e222fa314da382d3c3f96ee992aa 2023-04-23 05:56:01,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5938e222fa314da382d3c3f96ee992aa, entries=150, sequenceid=434, filesize=11.9 K 2023-04-23 05:56:01,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/1879b28da4d840bf84695e6027c35473 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1879b28da4d840bf84695e6027c35473 2023-04-23 05:56:01,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1879b28da4d840bf84695e6027c35473, entries=150, sequenceid=434, filesize=11.9 K 2023-04-23 05:56:01,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1f4745474d724f7886382245dcced426 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1f4745474d724f7886382245dcced426 2023-04-23 05:56:01,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1f4745474d724f7886382245dcced426, entries=150, sequenceid=434, filesize=11.9 K 2023-04-23 05:56:01,073 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~256.13 KB/262272, currentSize=87.22 KB/89310 for 70eda440525723289f2daf7f6251db0d in 878ms, sequenceid=434, compaction requested=true 2023-04-23 05:56:01,073 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:01,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:01,073 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:01,073 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:01,073 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:01,074 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:01,074 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:01,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:01,075 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:01,075 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:56:01,076 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:01,076 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9ae92832623148a8bd1f1f3b1116fe48, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/14557c4db40145ada919d4c94050af03, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5938e222fa314da382d3c3f96ee992aa] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=35.8 K 2023-04-23 05:56:01,076 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:01,076 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:56:01,076 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:01,076 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c7a93e6df8d64dc2b7c0bd13007dee14, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/b6ce4f6aecbd4806b96e5ee5766ae09f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1879b28da4d840bf84695e6027c35473] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=35.8 K 2023-04-23 05:56:01,078 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 9ae92832623148a8bd1f1f3b1116fe48, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682229357803 2023-04-23 05:56:01,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:01,079 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c7a93e6df8d64dc2b7c0bd13007dee14, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682229357803 2023-04-23 05:56:01,079 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 14557c4db40145ada919d4c94050af03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682229358545 2023-04-23 05:56:01,080 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b6ce4f6aecbd4806b96e5ee5766ae09f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682229358545 2023-04-23 05:56:01,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:01,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:01,080 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1879b28da4d840bf84695e6027c35473, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682229360176 2023-04-23 05:56:01,080 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:01,080 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 5938e222fa314da382d3c3f96ee992aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682229360176 2023-04-23 05:56:01,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:01,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:01,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:01,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:01,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:01,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:01,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:01,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:01,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:01,084 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:01,084 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:01,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:01,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:01,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:01,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:01,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:01,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:01,096 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:01,096 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:01,096 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:01,103 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#436 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:01,109 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:01,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:01,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:01,112 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-23 05:56:01,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:01,112 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:01,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:01,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:01,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:01,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:01,146 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/6e8af13676714adfa3c4534063817056 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e8af13676714adfa3c4534063817056 2023-04-23 05:56:01,153 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into 6e8af13676714adfa3c4534063817056(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:01,153 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:01,154 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=13, startTime=1682229361073; duration=0sec 2023-04-23 05:56:01,154 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:01,154 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:01,155 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:01,155 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/C is initiating minor compaction (all files) 2023-04-23 05:56:01,155 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/C in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:01,155 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2e31d35cb1134ea39d1aa9fdf1e4d620, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d5add1b081e04375868504bd83e55b45, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1f4745474d724f7886382245dcced426] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=35.8 K 2023-04-23 05:56:01,156 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 2e31d35cb1134ea39d1aa9fdf1e4d620, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1682229357803 2023-04-23 05:56:01,156 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d5add1b081e04375868504bd83e55b45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1682229358545 2023-04-23 05:56:01,157 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1f4745474d724f7886382245dcced426, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682229360176 2023-04-23 05:56:01,168 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#C#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:01,176 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/dfb2da64b5a0452faddd6d9b563b8d72 2023-04-23 05:56:01,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:01,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:01,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:01,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229421411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:01,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:01,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229421412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:01,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:01,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229421516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:01,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:01,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229421519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:01,537 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/b4a69546892f4d9e8ffed6956633b766 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/b4a69546892f4d9e8ffed6956633b766 2023-04-23 05:56:01,541 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into b4a69546892f4d9e8ffed6956633b766(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:01,541 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:01,541 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=13, startTime=1682229361073; duration=0sec 2023-04-23 05:56:01,541 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:01,600 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/9decc9a7bc0c4f11912a36126a3ab20e 2023-04-23 05:56:01,603 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/f80b9669fa7042539a6e6f05a4bb713e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/f80b9669fa7042539a6e6f05a4bb713e 2023-04-23 05:56:01,609 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/C of 70eda440525723289f2daf7f6251db0d into f80b9669fa7042539a6e6f05a4bb713e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:01,609 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:01,609 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/C, priority=13, startTime=1682229361074; duration=0sec 2023-04-23 05:56:01,609 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:01,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:01,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229421721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:01,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:01,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229421723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,011 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1eb1021e5bce4101b82534845feb0e67 2023-04-23 05:56:02,016 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/dfb2da64b5a0452faddd6d9b563b8d72 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dfb2da64b5a0452faddd6d9b563b8d72 2023-04-23 05:56:02,019 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dfb2da64b5a0452faddd6d9b563b8d72, entries=150, sequenceid=450, filesize=11.9 K 2023-04-23 05:56:02,020 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/9decc9a7bc0c4f11912a36126a3ab20e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9decc9a7bc0c4f11912a36126a3ab20e 2023-04-23 05:56:02,023 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9decc9a7bc0c4f11912a36126a3ab20e, entries=150, sequenceid=450, filesize=11.9 K 2023-04-23 05:56:02,024 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1eb1021e5bce4101b82534845feb0e67 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1eb1021e5bce4101b82534845feb0e67 2023-04-23 05:56:02,027 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1eb1021e5bce4101b82534845feb0e67, entries=150, sequenceid=450, filesize=11.9 K 2023-04-23 05:56:02,028 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 70eda440525723289f2daf7f6251db0d in 916ms, sequenceid=450, compaction requested=false 2023-04-23 05:56:02,028 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:02,028 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:02,030 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:02,031 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-23 05:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:02,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:02,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:02,033 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:02,033 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e1da6eb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:02,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:02,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:02,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:02,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:02,034 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:02,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:02,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:02,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:02,035 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:02,043 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:02,043 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:02,043 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:02,043 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:02,043 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:02,046 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,046 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,046 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:02,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:02,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,050 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:02,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,051 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:02,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,051 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:02,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:02,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229422062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:02,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229422063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,151 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:02,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:02,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:02,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:02,155 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:02,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:02,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:02,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:02,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:02,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:02,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:02,158 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:02,158 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:02,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:02,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:02,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:02,161 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:02,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65760ab9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:02,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:02,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,168 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,169 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,169 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:02,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:02,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:02,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:02,170 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:02,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:02,173 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:02,173 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:02,173 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:02,173 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:02,173 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:02,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:02,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229422167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,174 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,174 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,174 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229422172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,176 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,177 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,182 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,182 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,182 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,182 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,182 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,182 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,182 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,182 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,182 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:02,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:02,182 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,182 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:02,183 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:02,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:02,283 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:02,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:02,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:02,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:02,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:02,286 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:02,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:02,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:02,287 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,287 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:02,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:02,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:02,290 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,290 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:02,290 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:02,290 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:02,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,292 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:02,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@584275cc[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:02,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:02,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,297 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,298 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:02,298 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:02,298 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:02,298 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,298 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:02,299 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:02,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:02,302 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:02,302 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:02,302 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:02,302 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:02,302 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:02,303 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,303 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,303 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,308 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,311 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,311 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,311 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,311 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:02,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:02,311 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,312 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,312 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,312 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,312 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,312 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,312 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:02,312 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:02,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:02,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229422376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:02,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229422376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,412 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:02,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:02,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:02,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:02,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:02,416 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:02,416 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,416 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:02,416 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:02,416 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,416 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,420 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,420 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,420 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,420 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:02,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:02,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:02,422 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,422 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:02,422 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:02,422 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:02,423 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,425 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2128a4ba[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:02,427 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:02,427 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,428 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,428 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:02,428 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:02,428 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:02,428 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,428 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:02,431 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:02,433 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:02,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:02,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:02,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:02,434 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:02,436 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,436 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,436 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,437 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,440 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:02,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:02,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,440 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:02,440 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:02,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/35e43ea67b4a46b283e4bd20d0f0cf2c 2023-04-23 05:56:02,541 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:02,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:02,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:02,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:02,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:02,544 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:02,544 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,544 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:02,544 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:02,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,546 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,546 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,546 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,546 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:02,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:02,547 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:02,547 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,547 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:02,548 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,548 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:02,548 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:02,548 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:02,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:02,549 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,549 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:02,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:02,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42e1748a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:02,551 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:02,552 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,554 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:02,554 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,554 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,556 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:02,559 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:02,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:02,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:02,559 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:02,559 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:02,561 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,561 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,561 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,562 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,563 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,569 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:02,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:02,569 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:02,569 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:02,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,570 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,669 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:02,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:02,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:02,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:02,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:02,673 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:02,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:02,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:02,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,675 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,675 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:02,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:02,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:02,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:02,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:02,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:02,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:02,677 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,677 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:02,677 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:02,678 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:02,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:02,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:02,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:02,679 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:02,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17178dad[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:02,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:02,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,684 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,684 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:02,684 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:02,684 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:02,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:02,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,685 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:02,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:02,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:02,686 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:02,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:02,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:02,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:02,686 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:02,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:02,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229422682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:02,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229422682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,688 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,689 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,691 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,694 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,694 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,694 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,695 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:02,795 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:02,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:02,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:02,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:02,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:02,798 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:02,798 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,798 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:02,798 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:02,799 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,799 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,801 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,801 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,801 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,801 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:02,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:02,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:02,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:02,802 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:02,802 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,802 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:02,803 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:02,803 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,803 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:02,803 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:02,803 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,803 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:02,803 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:02,804 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,804 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:02,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c307696[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:02,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:02,807 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:02,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:02,808 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,808 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:02,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:02,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:02,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:02,812 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:02,813 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:02,813 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:02,813 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:02,813 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:02,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,815 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:02,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:02,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,818 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,820 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:02,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:02,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:02,819 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,820 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:02,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:02,820 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:02,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:02,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:02,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/88620571ddfe4e919e10a614f7067c57 2023-04-23 05:56:02,869 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/b4ceef151a2f481f9e7986d531bff6a1 2023-04-23 05:56:02,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/35e43ea67b4a46b283e4bd20d0f0cf2c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/35e43ea67b4a46b283e4bd20d0f0cf2c 2023-04-23 05:56:02,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/35e43ea67b4a46b283e4bd20d0f0cf2c, entries=200, sequenceid=471, filesize=14.3 K 2023-04-23 05:56:02,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/88620571ddfe4e919e10a614f7067c57 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/88620571ddfe4e919e10a614f7067c57 2023-04-23 05:56:02,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/88620571ddfe4e919e10a614f7067c57, entries=150, sequenceid=471, filesize=11.9 K 2023-04-23 05:56:02,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/b4ceef151a2f481f9e7986d531bff6a1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b4ceef151a2f481f9e7986d531bff6a1 2023-04-23 05:56:02,887 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b4ceef151a2f481f9e7986d531bff6a1, entries=150, sequenceid=471, filesize=11.9 K 2023-04-23 05:56:02,888 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 70eda440525723289f2daf7f6251db0d in 857ms, sequenceid=471, compaction requested=true 2023-04-23 05:56:02,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:02,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:02,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:02,888 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:02,888 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:02,888 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:02,890 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:02,890 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:02,890 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:56:02,890 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:56:02,890 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,890 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,890 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/b4a69546892f4d9e8ffed6956633b766, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dfb2da64b5a0452faddd6d9b563b8d72, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/35e43ea67b4a46b283e4bd20d0f0cf2c] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=38.6 K 2023-04-23 05:56:02,890 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e8af13676714adfa3c4534063817056, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9decc9a7bc0c4f11912a36126a3ab20e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/88620571ddfe4e919e10a614f7067c57] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=36.2 K 2023-04-23 05:56:02,891 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting b4a69546892f4d9e8ffed6956633b766, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682229360176 2023-04-23 05:56:02,891 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 6e8af13676714adfa3c4534063817056, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682229360176 2023-04-23 05:56:02,891 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting dfb2da64b5a0452faddd6d9b563b8d72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1682229360224 2023-04-23 05:56:02,891 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9decc9a7bc0c4f11912a36126a3ab20e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1682229360224 2023-04-23 05:56:02,891 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 35e43ea67b4a46b283e4bd20d0f0cf2c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682229361387 2023-04-23 05:56:02,892 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 88620571ddfe4e919e10a614f7067c57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682229361387 2023-04-23 05:56:02,898 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#445 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:02,899 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#446 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:02,914 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/7af14f5582d84155b52a5ecee51de320 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7af14f5582d84155b52a5ecee51de320 2023-04-23 05:56:02,916 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/c21124dccb304b9baec101d952a4fd19 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/c21124dccb304b9baec101d952a4fd19 2023-04-23 05:56:02,919 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into 7af14f5582d84155b52a5ecee51de320(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:02,919 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:02,919 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=13, startTime=1682229362888; duration=0sec 2023-04-23 05:56:02,919 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:02,919 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:02,920 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into c21124dccb304b9baec101d952a4fd19(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:02,920 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:02,920 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=13, startTime=1682229362888; duration=0sec 2023-04-23 05:56:02,920 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:02,920 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:02,920 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:02,920 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/C is initiating minor compaction (all files) 2023-04-23 05:56:02,920 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/C in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,921 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/f80b9669fa7042539a6e6f05a4bb713e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1eb1021e5bce4101b82534845feb0e67, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b4ceef151a2f481f9e7986d531bff6a1] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=36.2 K 2023-04-23 05:56:02,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:02,921 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f80b9669fa7042539a6e6f05a4bb713e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=434, earliestPutTs=1682229360176 2023-04-23 05:56:02,922 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1eb1021e5bce4101b82534845feb0e67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1682229360224 2023-04-23 05:56:02,922 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b4ceef151a2f481f9e7986d531bff6a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682229361387 2023-04-23 05:56:02,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:02,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:02,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:02,923 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:02,923 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,923 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:02,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:02,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,925 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:02,925 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:02,926 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:02,926 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,927 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#C#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:02,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:02,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:02,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:02,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:02,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:02,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:02,928 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:02,933 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:02,933 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:02,933 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:02,933 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:02,933 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-23 05:56:02,933 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:02,934 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:02,934 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:02,934 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:02,934 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:02,934 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:02,942 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/c69aa29aea204143a32e9da29b1039bf as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c69aa29aea204143a32e9da29b1039bf 2023-04-23 05:56:02,947 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/C of 70eda440525723289f2daf7f6251db0d into c69aa29aea204143a32e9da29b1039bf(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:02,947 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:02,947 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/C, priority=13, startTime=1682229362888; duration=0sec 2023-04-23 05:56:02,947 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:02,947 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/3650dabe0d3c4448b6a5138cf4f48be7 2023-04-23 05:56:03,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:03,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:03,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229423239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229423240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229423343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229423346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,358 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/dc942ae2b1c8498d98bc5c816fc1a178 2023-04-23 05:56:03,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229423547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229423550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,772 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/6b193562598d4653aa28e0a6d31ac3c8 2023-04-23 05:56:03,775 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/3650dabe0d3c4448b6a5138cf4f48be7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/3650dabe0d3c4448b6a5138cf4f48be7 2023-04-23 05:56:03,779 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/3650dabe0d3c4448b6a5138cf4f48be7, entries=150, sequenceid=487, filesize=11.9 K 2023-04-23 05:56:03,779 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/dc942ae2b1c8498d98bc5c816fc1a178 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/dc942ae2b1c8498d98bc5c816fc1a178 2023-04-23 05:56:03,783 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/dc942ae2b1c8498d98bc5c816fc1a178, entries=150, sequenceid=487, filesize=11.9 K 2023-04-23 05:56:03,784 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/6b193562598d4653aa28e0a6d31ac3c8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/6b193562598d4653aa28e0a6d31ac3c8 2023-04-23 05:56:03,787 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/6b193562598d4653aa28e0a6d31ac3c8, entries=150, sequenceid=487, filesize=11.9 K 2023-04-23 05:56:03,788 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 70eda440525723289f2daf7f6251db0d in 855ms, sequenceid=487, compaction requested=false 2023-04-23 05:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:03,788 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:03,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:03,790 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,790 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:03,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:03,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@385914de[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:03,793 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:03,793 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,795 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:03,795 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,795 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:03,795 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:03,797 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:03,797 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:03,797 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:03,797 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:03,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:03,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:03,801 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:03,801 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:03,801 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:03,801 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:03,801 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:03,802 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,802 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:03,802 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,803 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:03,804 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,807 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:03,807 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:03,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:03,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:03,807 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,807 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:03,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:03,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:03,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:03,808 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:03,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,808 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:03,808 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:03,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:03,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:03,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-23 05:56:03,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:03,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:03,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:03,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:03,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:03,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:03,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/1c7acc7df28f4b40bddfd65cd9712133 2023-04-23 05:56:03,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229423880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229423885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,908 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:03,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:03,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:03,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:03,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:03,911 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:03,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:03,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:03,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:03,912 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,912 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:03,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,914 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:03,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:03,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:03,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:03,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:03,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:03,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:03,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:03,915 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:03,915 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:03,915 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:03,915 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:03,915 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:03,916 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:03,916 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:03,916 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,916 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:03,916 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:03,922 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:03,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:03,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d5627b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:03,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:03,924 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,927 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,927 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:03,927 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:03,927 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:03,927 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,927 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:03,929 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:03,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:03,929 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:03,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:03,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:03,931 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:03,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:03,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:03,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:03,931 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:03,933 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,933 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:03,934 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,933 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:03,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:03,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:03,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:03,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:03,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:03,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:03,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:03,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:03,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:03,938 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:03,938 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:03,938 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:03,939 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:03,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:03,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229423990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:03,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:03,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229423991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,039 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:04,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:04,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:04,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:04,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:04,042 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:04,042 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:04,042 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:04,042 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:04,043 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,043 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,044 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,044 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:04,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:04,044 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,044 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:04,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:04,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:04,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:04,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:04,045 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:04,046 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,046 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:04,046 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:04,046 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:04,046 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:04,047 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,047 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:04,048 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,048 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:04,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a04f309[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:04,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:04,049 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,050 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:04,050 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:04,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:04,052 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:04,052 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:04,053 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:04,053 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:04,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:04,055 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:04,055 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:04,055 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:04,055 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:04,055 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:04,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,056 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:04,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,057 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,058 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:04,060 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:04,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:04,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,062 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:04,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,062 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:04,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:04,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:04,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:04,063 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:04,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:04,063 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:04,163 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:04,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:04,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:04,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:04,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:04,166 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:04,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:04,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:04,166 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:04,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,167 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,168 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:04,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:04,168 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,168 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,168 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:04,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:04,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:04,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:04,169 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:04,169 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,170 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:04,170 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:04,170 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:04,170 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:04,172 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:04,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:04,173 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,173 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:04,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dfaa19[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:04,174 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:04,174 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,175 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:04,175 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:04,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,175 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:04,176 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,176 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:04,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:04,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:04,179 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:04,179 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:04,179 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:04,179 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:04,179 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:04,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:04,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:04,181 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,181 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:04,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,182 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,192 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:04,192 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:04,192 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:04,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:04,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:04,192 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:04,192 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,192 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,192 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,192 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,193 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:04,193 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:04,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:04,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229424193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229424194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/f57bd9a049e043dcaecf60fc927bad80 2023-04-23 05:56:04,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/2237bfee41974e6ea3d419a7755db652 2023-04-23 05:56:04,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/1c7acc7df28f4b40bddfd65cd9712133 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1c7acc7df28f4b40bddfd65cd9712133 2023-04-23 05:56:04,293 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:04,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:04,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1c7acc7df28f4b40bddfd65cd9712133, entries=200, sequenceid=507, filesize=14.3 K 2023-04-23 05:56:04,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:04,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/f57bd9a049e043dcaecf60fc927bad80 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f57bd9a049e043dcaecf60fc927bad80 2023-04-23 05:56:04,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:04,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:04,297 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:04,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:04,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:04,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:04,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,298 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:04,298 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:04,299 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,299 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:04,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:04,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:04,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:04,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:04,300 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:04,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f57bd9a049e043dcaecf60fc927bad80, entries=150, sequenceid=507, filesize=11.9 K 2023-04-23 05:56:04,300 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,301 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-23 05:56:04,301 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:04,301 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-23 05:56:04,301 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:04,301 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:04,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:04,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:04,301 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:04,302 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,302 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:04,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,303 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:04,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/2237bfee41974e6ea3d419a7755db652 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2237bfee41974e6ea3d419a7755db652 2023-04-23 05:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:04,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:04,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7550a198[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:04,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:04,304 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2237bfee41974e6ea3d419a7755db652, entries=150, sequenceid=507, filesize=11.9 K 2023-04-23 05:56:04,307 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 70eda440525723289f2daf7f6251db0d in 449ms, sequenceid=507, compaction requested=true 2023-04-23 05:56:04,307 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,307 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:04,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:04,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:04,307 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:04,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-23 05:56:04,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:04,307 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,307 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:04,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:04,307 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:04,307 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:04,308 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:04,308 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:56:04,308 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:04,308 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/c21124dccb304b9baec101d952a4fd19, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/3650dabe0d3c4448b6a5138cf4f48be7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1c7acc7df28f4b40bddfd65cd9712133] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=38.7 K 2023-04-23 05:56:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:04,308 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:04,309 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:04,309 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:56:04,309 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,309 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c21124dccb304b9baec101d952a4fd19, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682229361387 2023-04-23 05:56:04,309 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7af14f5582d84155b52a5ecee51de320, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/dc942ae2b1c8498d98bc5c816fc1a178, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f57bd9a049e043dcaecf60fc927bad80] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=36.3 K 2023-04-23 05:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:04,309 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 3650dabe0d3c4448b6a5138cf4f48be7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682229362043 2023-04-23 05:56:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:04,310 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7af14f5582d84155b52a5ecee51de320, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682229361387 2023-04-23 05:56:04,310 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 1c7acc7df28f4b40bddfd65cd9712133, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1682229363232 2023-04-23 05:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,311 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting dc942ae2b1c8498d98bc5c816fc1a178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682229362043 2023-04-23 05:56:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,311 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f57bd9a049e043dcaecf60fc927bad80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1682229363232 2023-04-23 05:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:04,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:04,313 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:04,313 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:04,313 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:04,313 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:04,313 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:04,314 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,314 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:04,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,315 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:04,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,315 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,319 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#454 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:04,322 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:04,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:04,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:04,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:04,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:04,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:04,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:04,326 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:04,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:04,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,326 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:04,326 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:04,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:04,329 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#455 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:04,336 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/dd1efa6fdd334062a88f86e68d50ee0b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dd1efa6fdd334062a88f86e68d50ee0b 2023-04-23 05:56:04,339 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into dd1efa6fdd334062a88f86e68d50ee0b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:04,340 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:04,340 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=13, startTime=1682229364307; duration=0sec 2023-04-23 05:56:04,340 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:04,340 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:04,341 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:04,341 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/C is initiating minor compaction (all files) 2023-04-23 05:56:04,341 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/C in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,341 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c69aa29aea204143a32e9da29b1039bf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/6b193562598d4653aa28e0a6d31ac3c8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2237bfee41974e6ea3d419a7755db652] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=36.3 K 2023-04-23 05:56:04,341 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c69aa29aea204143a32e9da29b1039bf, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682229361387 2023-04-23 05:56:04,342 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 6b193562598d4653aa28e0a6d31ac3c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1682229362043 2023-04-23 05:56:04,342 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 2237bfee41974e6ea3d419a7755db652, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1682229363232 2023-04-23 05:56:04,350 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/0d1dca50b67f405d8d9070a693213bc3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/0d1dca50b67f405d8d9070a693213bc3 2023-04-23 05:56:04,351 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#C#compaction#456 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:04,354 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into 0d1dca50b67f405d8d9070a693213bc3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:04,354 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:04,354 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=13, startTime=1682229364307; duration=0sec 2023-04-23 05:56:04,354 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:04,362 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/78dc3ccd5a2c470c86afe967129d1664 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/78dc3ccd5a2c470c86afe967129d1664 2023-04-23 05:56:04,370 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/C of 70eda440525723289f2daf7f6251db0d into 78dc3ccd5a2c470c86afe967129d1664(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:04,370 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:04,370 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/C, priority=13, startTime=1682229364307; duration=0sec 2023-04-23 05:56:04,370 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:04,427 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:04,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:04,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:04,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:04,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:04,430 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:04,430 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:04,430 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:04,430 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:04,430 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,431 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:04,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:04,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,433 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:04,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:04,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:04,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:04,435 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:04,436 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:04,436 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:04,436 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-23 05:56:04,436 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:04,436 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:04,436 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:04,437 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:04,437 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:04,437 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:04,445 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/e1630bbc8a3248048d140304d14c80d4 2023-04-23 05:56:04,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:04,501 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:04,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229424539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229424542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229424643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229424647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229424849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229424856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,862 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/6e7d41c04d8541aca4da3d61265a379d 2023-04-23 05:56:04,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38336 deadline: 1682229424864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,870 DEBUG [Thread-3033] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:04,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38388 deadline: 1682229424874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:04,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:38380 deadline: 1682229424876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:04,880 DEBUG [Thread-3035] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:04,881 DEBUG [Thread-3031] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:05,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:05,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229425154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:05,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229425158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,275 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/58d18a164549447fbce89d46a8fdd505 2023-04-23 05:56:05,279 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/e1630bbc8a3248048d140304d14c80d4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e1630bbc8a3248048d140304d14c80d4 2023-04-23 05:56:05,284 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e1630bbc8a3248048d140304d14c80d4, entries=150, sequenceid=523, filesize=11.9 K 2023-04-23 05:56:05,284 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/6e7d41c04d8541aca4da3d61265a379d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e7d41c04d8541aca4da3d61265a379d 2023-04-23 05:56:05,287 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e7d41c04d8541aca4da3d61265a379d, entries=150, sequenceid=523, filesize=11.9 K 2023-04-23 05:56:05,288 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/58d18a164549447fbce89d46a8fdd505 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/58d18a164549447fbce89d46a8fdd505 2023-04-23 05:56:05,291 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/58d18a164549447fbce89d46a8fdd505, entries=150, sequenceid=523, filesize=11.9 K 2023-04-23 05:56:05,291 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 70eda440525723289f2daf7f6251db0d in 855ms, sequenceid=523, compaction requested=false 2023-04-23 05:56:05,291 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:05,291 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:05,291 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:05,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:05,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,293 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,294 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,294 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:05,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f4cc3f3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:05,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:05,296 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,297 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,297 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:05,297 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:05,297 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:05,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,297 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:05,298 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,298 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:05,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,299 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:05,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:05,301 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:05,301 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:05,301 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:05,301 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:05,301 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:05,303 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,304 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,304 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,306 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:05,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:05,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:05,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:05,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,308 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,308 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:05,309 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,309 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:05,309 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:05,409 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:05,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:05,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:05,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:05,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:05,413 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:05,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:05,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:05,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:05,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,414 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:05,414 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:05,415 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,415 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:05,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:05,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:05,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:05,416 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:05,416 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:05,416 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-23 05:56:05,417 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:05,417 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:05,417 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:05,417 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:05,417 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:05,417 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:05,426 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/f89ba522e3cf4f4281c73bfd33715414 2023-04-23 05:56:05,439 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/a030038c4eca44debfb18d9576dc354d 2023-04-23 05:56:05,469 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/c27bc7c7d27246c7b82b6c5aff87ea17 2023-04-23 05:56:05,474 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/f89ba522e3cf4f4281c73bfd33715414 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f89ba522e3cf4f4281c73bfd33715414 2023-04-23 05:56:05,481 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f89ba522e3cf4f4281c73bfd33715414, entries=150, sequenceid=542, filesize=11.9 K 2023-04-23 05:56:05,482 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/a030038c4eca44debfb18d9576dc354d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a030038c4eca44debfb18d9576dc354d 2023-04-23 05:56:05,489 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a030038c4eca44debfb18d9576dc354d, entries=150, sequenceid=542, filesize=11.9 K 2023-04-23 05:56:05,489 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/c27bc7c7d27246c7b82b6c5aff87ea17 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c27bc7c7d27246c7b82b6c5aff87ea17 2023-04-23 05:56:05,492 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c27bc7c7d27246c7b82b6c5aff87ea17, entries=150, sequenceid=542, filesize=11.9 K 2023-04-23 05:56:05,493 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 70eda440525723289f2daf7f6251db0d in 77ms, sequenceid=542, compaction requested=true 2023-04-23 05:56:05,493 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:05,493 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:05,493 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:05,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:05,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:05,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:05,495 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,495 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:05,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@522de1a8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:05,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:05,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,499 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:05,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:05,500 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:05,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,500 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:05,501 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:05,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,501 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:05,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:05,504 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:05,504 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:05,504 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:05,504 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:05,504 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:05,505 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,505 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,506 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,507 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,509 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:05,509 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,509 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,509 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,509 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:05,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:05,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:05,509 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,510 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:05,510 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:05,510 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,510 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,510 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,510 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:05,610 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:05,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:05,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:05,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:05,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:05,614 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:05,614 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:05,614 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:05,614 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:05,615 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,615 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,618 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:05,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:05,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:05,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:05,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:05,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:05,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:05,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:05,620 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:05,620 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:05,620 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:05,620 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:05,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:05,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,622 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,623 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,623 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:05,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ada1696[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:05,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:05,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,627 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:05,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:05,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:05,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:05,629 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:05,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:05,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:05,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:05,633 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:05,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:05,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:05,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:05,633 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:05,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,639 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,641 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:05,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:05,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:05,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,643 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:05,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:05,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:05,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,644 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,644 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:05,644 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:05,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:05,676 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:56:05,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:05,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:05,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:05,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:05,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:05,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:05,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/fa9a5513a7a74d4f9306b8a72dd40852 2023-04-23 05:56:05,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682229365706, max timestamp=1682229365734Num uniques 859; 2023-04-23 05:56:05,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-23 05:56:05,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682229365706, max timestamp=1682229365734Num uniques 859; 2023-04-23 05:56:05,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-23 05:56:05,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682229365706, max timestamp=1682229365734Num uniques 859; 2023-04-23 05:56:05,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-23 05:56:05,744 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:05,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:05,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:05,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:05,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:05,748 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:05,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:05,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:05,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:05,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:05,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:05,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:05,749 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:05,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:05,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:05,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:05,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:05,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:05,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:05,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:05,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:05,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:05,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:05,751 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:05,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:05,754 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:05,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:05,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af63e34[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:05,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,756 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:05,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:05,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:05,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:05,757 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:05,758 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,758 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:05,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:05,760 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:05,760 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:05,760 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:05,760 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:05,760 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:05,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:05,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:05,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:05,773 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:05,773 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:05,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:05,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229425815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:05,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229425821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,873 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:05,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:05,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:05,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:05,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:05,877 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:05,877 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:05,877 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:05,877 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:05,877 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,877 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:05,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:05,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:05,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:05,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:05,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:05,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:05,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:05,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:05,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:05,880 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:05,880 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:05,880 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:05,880 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:05,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:05,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:05,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:05,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:05,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,882 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,882 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,882 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:05,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13ec2dc2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:05,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:05,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,885 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:05,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:05,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:05,885 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,885 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:05,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:05,887 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:05,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:05,890 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:05,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:05,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:05,890 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:05,890 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:05,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:05,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:05,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,894 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:05,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:05,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:05,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:05,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:05,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:05,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:05,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:05,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:05,897 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:05,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:05,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:05,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:05,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:05,897 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:05,897 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:05,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:05,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229425924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:05,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229425929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:05,997 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:05,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:06,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:06,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:06,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:06,001 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:06,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:06,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:06,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:06,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:06,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:06,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:06,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:06,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:06,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:06,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:06,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:06,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:06,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:06,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:06,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:06,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:06,004 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:06,004 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:06,004 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:06,004 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:06,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:06,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:06,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:06,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:06,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,007 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:06,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:06,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:06,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:06,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:06,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:06,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e71beb6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:06,011 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:06,011 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:06,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:06,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:06,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:06,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:06,012 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:06,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:06,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,013 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:06,013 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:06,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:06,013 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:06,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:06,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:06,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:06,017 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:06,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:06,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:06,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:06,017 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:06,019 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,019 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:06,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,019 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:06,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:06,020 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,021 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:06,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,026 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,026 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:06,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:06,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:06,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:06,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:06,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:06,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:06,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:06,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:06,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:06,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:06,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:06,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:06,028 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:06,028 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:06,095 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/f83814602ba04ccba6fc72ad510b250c 2023-04-23 05:56:06,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/d1dc23237718441f88ba61a3487af088 2023-04-23 05:56:06,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/fa9a5513a7a74d4f9306b8a72dd40852 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/fa9a5513a7a74d4f9306b8a72dd40852 2023-04-23 05:56:06,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/fa9a5513a7a74d4f9306b8a72dd40852, entries=200, sequenceid=553, filesize=14.3 K 2023-04-23 05:56:06,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/f83814602ba04ccba6fc72ad510b250c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f83814602ba04ccba6fc72ad510b250c 2023-04-23 05:56:06,125 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f83814602ba04ccba6fc72ad510b250c, entries=150, sequenceid=553, filesize=11.9 K 2023-04-23 05:56:06,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/d1dc23237718441f88ba61a3487af088 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d1dc23237718441f88ba61a3487af088 2023-04-23 05:56:06,128 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:06,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:06,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d1dc23237718441f88ba61a3487af088, entries=150, sequenceid=553, filesize=11.9 K 2023-04-23 05:56:06,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 70eda440525723289f2daf7f6251db0d in 454ms, sequenceid=553, compaction requested=true 2023-04-23 05:56:06,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:06,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:06,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:06,130 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:06,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:06,130 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:06,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:06,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:06,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:06,132 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:06,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:06,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:06,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:06,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,132 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51934 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:06,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:06,133 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:56:06,133 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:06,133 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dd1efa6fdd334062a88f86e68d50ee0b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e1630bbc8a3248048d140304d14c80d4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f89ba522e3cf4f4281c73bfd33715414, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/fa9a5513a7a74d4f9306b8a72dd40852] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=50.7 K 2023-04-23 05:56:06,133 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:06,133 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:56:06,133 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:06,133 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/0d1dca50b67f405d8d9070a693213bc3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e7d41c04d8541aca4da3d61265a379d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a030038c4eca44debfb18d9576dc354d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f83814602ba04ccba6fc72ad510b250c] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=48.3 K 2023-04-23 05:56:06,134 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:06,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:06,134 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 0d1dca50b67f405d8d9070a693213bc3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1682229363232 2023-04-23 05:56:06,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:06,134 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting dd1efa6fdd334062a88f86e68d50ee0b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1682229363232 2023-04-23 05:56:06,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:06,135 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 6e7d41c04d8541aca4da3d61265a379d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682229363877 2023-04-23 05:56:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:06,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:06,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:06,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:06,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:06,135 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a030038c4eca44debfb18d9576dc354d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682229364533 2023-04-23 05:56:06,135 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:06,135 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:06,135 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:06,136 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e1630bbc8a3248048d140304d14c80d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682229363877 2023-04-23 05:56:06,136 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:06,136 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:06,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,136 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f83814602ba04ccba6fc72ad510b250c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1682229365670 2023-04-23 05:56:06,136 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:06,136 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:06,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:06,136 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-23 05:56:06,137 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:06,137 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:06,137 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:06,137 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:06,137 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:06,137 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:06,137 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting f89ba522e3cf4f4281c73bfd33715414, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682229364533 2023-04-23 05:56:06,138 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting fa9a5513a7a74d4f9306b8a72dd40852, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1682229365669 2023-04-23 05:56:06,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:06,140 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:06,149 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#467 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:06,151 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#468 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:06,160 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/750850e28ce24cc584cac134feea48b6 2023-04-23 05:56:06,169 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/2896ed3ccef44ac29428a734410e65e9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/2896ed3ccef44ac29428a734410e65e9 2023-04-23 05:56:06,173 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into 2896ed3ccef44ac29428a734410e65e9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:06,173 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:06,173 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=12, startTime=1682229366130; duration=0sec 2023-04-23 05:56:06,173 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:06,173 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:06,174 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:06,174 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/C is initiating minor compaction (all files) 2023-04-23 05:56:06,174 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/C in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:06,174 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/78dc3ccd5a2c470c86afe967129d1664, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/58d18a164549447fbce89d46a8fdd505, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c27bc7c7d27246c7b82b6c5aff87ea17, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d1dc23237718441f88ba61a3487af088] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=48.3 K 2023-04-23 05:56:06,175 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 78dc3ccd5a2c470c86afe967129d1664, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1682229363232 2023-04-23 05:56:06,175 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 58d18a164549447fbce89d46a8fdd505, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682229363877 2023-04-23 05:56:06,175 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c27bc7c7d27246c7b82b6c5aff87ea17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1682229364533 2023-04-23 05:56:06,176 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting d1dc23237718441f88ba61a3487af088, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1682229365670 2023-04-23 05:56:06,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:06,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229426168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:06,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229426169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,187 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#C#compaction#470 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:06,205 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/591648664b0649518239675d5c4274dc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/591648664b0649518239675d5c4274dc 2023-04-23 05:56:06,212 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70eda440525723289f2daf7f6251db0d/C of 70eda440525723289f2daf7f6251db0d into 591648664b0649518239675d5c4274dc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:06,212 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:06,212 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/C, priority=12, startTime=1682229366130; duration=0sec 2023-04-23 05:56:06,212 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:06,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:06,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229426279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:06,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229426280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:06,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229426484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:06,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229426484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,567 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/3bc7cf5cb9944319b5ba0cfe9b5df332 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/3bc7cf5cb9944319b5ba0cfe9b5df332 2023-04-23 05:56:06,571 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into 3bc7cf5cb9944319b5ba0cfe9b5df332(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:06,571 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:06,571 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/8e65c3e2da5941039f070b1276a91347 2023-04-23 05:56:06,571 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=12, startTime=1682229366130; duration=0sec 2023-04-23 05:56:06,571 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:06,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:06,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229426790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:06,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229426790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:06,980 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/db41f9333e7a4e78817a73b72fae4b66 2023-04-23 05:56:06,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/750850e28ce24cc584cac134feea48b6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/750850e28ce24cc584cac134feea48b6 2023-04-23 05:56:06,990 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/750850e28ce24cc584cac134feea48b6, entries=150, sequenceid=588, filesize=11.9 K 2023-04-23 05:56:06,990 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/8e65c3e2da5941039f070b1276a91347 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/8e65c3e2da5941039f070b1276a91347 2023-04-23 05:56:06,994 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/8e65c3e2da5941039f070b1276a91347, entries=150, sequenceid=588, filesize=11.9 K 2023-04-23 05:56:06,995 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/db41f9333e7a4e78817a73b72fae4b66 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/db41f9333e7a4e78817a73b72fae4b66 2023-04-23 05:56:07,001 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/db41f9333e7a4e78817a73b72fae4b66, entries=150, sequenceid=588, filesize=11.9 K 2023-04-23 05:56:07,002 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 70eda440525723289f2daf7f6251db0d in 866ms, sequenceid=588, compaction requested=false 2023-04-23 05:56:07,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:07,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:07,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:07,003 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:07,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54657348[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:07,005 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:07,005 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,007 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,007 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:07,007 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:07,007 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:07,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:07,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:07,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:07,011 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:07,011 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:07,011 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:07,011 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:07,011 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:07,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,014 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:07,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:07,016 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,016 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:07,016 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:07,017 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,017 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,116 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:07,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:07,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:07,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:07,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:07,120 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:07,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:07,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:07,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,122 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,123 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,123 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:07,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:07,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:07,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:07,124 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:07,124 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,124 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:07,124 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:07,124 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:07,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:07,124 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:56:07,125 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:07,125 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:07,125 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:07,125 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:07,125 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:07,125 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:07,133 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/343ad97320434559a3b837607a231cb5 2023-04-23 05:56:07,145 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/f1d7ab62d51d4ea2b449a3610079bd96 2023-04-23 05:56:07,158 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/254e1c22b8044bbeb4a0505b0b94f500 2023-04-23 05:56:07,163 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/343ad97320434559a3b837607a231cb5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/343ad97320434559a3b837607a231cb5 2023-04-23 05:56:07,167 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/343ad97320434559a3b837607a231cb5, entries=150, sequenceid=601, filesize=11.9 K 2023-04-23 05:56:07,167 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/f1d7ab62d51d4ea2b449a3610079bd96 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f1d7ab62d51d4ea2b449a3610079bd96 2023-04-23 05:56:07,174 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f1d7ab62d51d4ea2b449a3610079bd96, entries=150, sequenceid=601, filesize=11.9 K 2023-04-23 05:56:07,175 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/254e1c22b8044bbeb4a0505b0b94f500 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/254e1c22b8044bbeb4a0505b0b94f500 2023-04-23 05:56:07,178 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/254e1c22b8044bbeb4a0505b0b94f500, entries=150, sequenceid=601, filesize=11.9 K 2023-04-23 05:56:07,179 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 70eda440525723289f2daf7f6251db0d in 55ms, sequenceid=601, compaction requested=true 2023-04-23 05:56:07,179 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:07,179 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:07,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:07,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:07,179 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:07,180 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,180 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,181 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,181 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:07,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:07,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f09e06e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:07,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:07,184 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,185 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,185 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:07,185 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:07,185 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:07,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,185 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:07,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:07,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,187 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:07,187 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:07,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:07,191 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:07,191 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:07,191 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:07,191 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:07,191 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:07,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,194 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,195 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,197 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,197 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:07,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:07,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:07,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,197 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,197 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,198 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,198 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:07,198 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,198 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,198 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:07,198 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,198 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,198 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,298 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:07,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:07,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:07,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:07,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:07,301 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:07,301 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,301 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:07,301 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:07,302 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,302 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,303 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,303 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,303 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,303 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:07,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:07,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:07,305 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:07,305 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,305 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:07,305 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:07,305 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:07,305 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,305 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:07,306 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-23 05:56:07,306 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:07,306 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:07,306 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:07,306 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:07,306 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:07,306 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:07,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:07,312 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:07,321 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/e4b2159aa978434eb2d6591e63092ddf 2023-04-23 05:56:07,332 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/c935dd35aedc41268ee014fd3844a75e 2023-04-23 05:56:07,344 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/06c1c1eef4b84c188a4dd7452985fe5b 2023-04-23 05:56:07,347 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/e4b2159aa978434eb2d6591e63092ddf as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e4b2159aa978434eb2d6591e63092ddf 2023-04-23 05:56:07,350 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e4b2159aa978434eb2d6591e63092ddf, entries=150, sequenceid=608, filesize=11.9 K 2023-04-23 05:56:07,351 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/c935dd35aedc41268ee014fd3844a75e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c935dd35aedc41268ee014fd3844a75e 2023-04-23 05:56:07,354 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c935dd35aedc41268ee014fd3844a75e, entries=150, sequenceid=608, filesize=11.9 K 2023-04-23 05:56:07,355 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/06c1c1eef4b84c188a4dd7452985fe5b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/06c1c1eef4b84c188a4dd7452985fe5b 2023-04-23 05:56:07,358 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/06c1c1eef4b84c188a4dd7452985fe5b, entries=150, sequenceid=608, filesize=11.9 K 2023-04-23 05:56:07,359 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=100.63 KB/103050 for 70eda440525723289f2daf7f6251db0d in 53ms, sequenceid=608, compaction requested=true 2023-04-23 05:56:07,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:07,359 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:07,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:07,360 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-23 05:56:07,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:07,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:07,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:07,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:07,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:07,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:07,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:07,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:07,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d795870[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:07,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:07,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,366 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:07,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:07,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:07,376 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:07,378 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,378 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:07,381 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:07,381 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:07,381 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:07,381 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:07,381 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:07,383 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,383 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,383 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,390 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,391 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:07,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:07,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,397 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:07,397 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:07,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:07,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229427424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:07,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229427424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,497 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:07,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:07,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:07,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:07,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:07,501 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:07,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:07,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:07,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,501 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,502 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,502 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:07,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:07,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:07,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,505 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:07,508 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,508 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:07,508 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:07,508 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:07,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:07,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,510 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e5ecb67[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:07,512 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:07,512 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:07,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:07,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:07,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:07,516 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:07,516 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,516 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:07,516 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:07,519 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:07,519 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:07,519 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:07,519 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:07,519 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:07,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:07,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:07,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:07,526 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,526 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:07,526 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:07,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:07,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229427531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:07,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229427531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,626 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:07,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:07,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:07,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:07,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:07,630 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:07,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:07,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:07,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,633 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:07,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:07,635 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,635 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:07,635 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:07,635 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:07,635 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:07,637 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,637 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,638 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,638 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:07,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11de0ad7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:07,639 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:07,639 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:07,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:07,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:07,640 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:07,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,641 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:07,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:07,644 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:07,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:07,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:07,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:07,644 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:07,646 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,647 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,647 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,652 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:07,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:07,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,652 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:07,652 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:07,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:07,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:07,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229427738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229427738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,752 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:07,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:07,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:07,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:07,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:07,756 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:07,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:07,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:07,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,757 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,758 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,758 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,758 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:07,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:07,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:07,759 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,760 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:07,760 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:07,760 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:07,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:07,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:07,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:07,761 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,761 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,762 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,762 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:07,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61a49212[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:07,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:07,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,766 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:07,766 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:07,766 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:07,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,766 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:07,767 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:07,767 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,767 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,767 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,770 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/19dff9dabf75402eba9897ca05e5c5ba 2023-04-23 05:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:07,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:07,770 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:07,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:07,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:07,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:07,770 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,772 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,774 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,775 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,777 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,777 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,777 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,777 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,777 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:07,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:07,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,777 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,778 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,778 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,778 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,778 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,778 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:07,778 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:07,878 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:07,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:07,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:07,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:07,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:07,882 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:07,882 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,882 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:07,882 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:07,882 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,882 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,885 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,885 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,886 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,886 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:07,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:07,887 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,887 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:07,887 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:07,887 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:07,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:07,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,892 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,892 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,892 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6619fc8e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:07,894 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:07,894 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,895 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:07,895 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:07,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:07,896 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:07,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,896 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,896 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:07,899 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:07,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:07,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:07,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:07,899 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:07,900 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,900 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,900 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:07,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,901 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,903 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:07,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:07,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:07,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:07,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:07,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:07,906 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:07,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:07,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:07,906 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:07,907 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:07,907 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:07,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,007 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:08,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:08,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:08,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:08,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:08,010 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:08,010 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,010 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:08,010 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:08,011 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,011 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:08,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:08,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:08,013 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:08,013 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,013 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:08,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:08,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:08,014 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:08,015 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,016 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,016 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,016 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:08,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8406d9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:08,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:08,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,018 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,018 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:08,018 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:08,018 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,018 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:08,018 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:08,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,024 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:08,027 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:08,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:08,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:08,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:08,027 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:08,028 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,028 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,028 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,033 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,033 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,033 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,033 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,033 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,033 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,033 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:08,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:08,033 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,034 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,034 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,034 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:08,034 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:08,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:08,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:08,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229428047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:08,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229428048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,134 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:08,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:08,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:08,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:08,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:08,137 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:08,138 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,138 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:08,138 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:08,138 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,138 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,139 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,140 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:08,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:08,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,141 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:08,142 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,142 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:08,142 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:08,142 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:08,142 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:08,145 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fed236c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:08,146 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:08,147 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,147 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,148 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:08,148 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:08,148 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:08,148 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,148 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:08,149 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,149 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:08,149 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,149 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:08,152 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:08,152 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:08,152 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:08,152 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:08,152 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:08,153 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,153 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,153 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,154 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,158 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:08,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:08,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,159 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:08,159 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:08,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:08,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,181 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/e3d36d3f901747fc947c0b3bac432d10 2023-04-23 05:56:08,259 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:08,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:08,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:08,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:08,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:08,263 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:08,263 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,263 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:08,263 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:08,263 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,263 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,264 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,264 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,265 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,265 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:08,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:08,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:08,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:08,266 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:08,266 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,266 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:08,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:08,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:08,267 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,267 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:08,267 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:08,267 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:08,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:08,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,267 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:08,268 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:08,269 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,269 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,269 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,269 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:08,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a9c1681[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:08,271 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:08,271 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,275 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:08,275 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:08,275 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:08,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:08,277 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:08,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:08,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:08,283 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:08,283 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:08,283 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:08,283 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:08,283 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:08,284 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,284 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,284 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:08,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:08,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,289 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:08,289 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:08,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,289 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:08,389 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:08,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:08,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:08,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:08,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:08,392 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:08,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:08,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:08,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,394 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,394 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:08,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:08,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:08,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:08,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:08,396 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,396 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:08,396 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:08,396 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:08,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:08,398 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,398 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,398 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:08,399 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:08,400 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:08,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c86cd0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:08,400 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:08,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:08,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:08,402 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,402 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:08,403 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,403 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:08,403 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,403 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:08,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:08,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:08,405 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:08,405 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:08,405 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:08,405 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:08,405 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:08,407 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,407 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,407 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,408 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,409 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,411 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,411 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,411 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,411 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:08,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:08,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:08,411 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,411 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,412 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:08,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,412 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:08,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,512 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:08,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:08,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:08,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:08,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:08,515 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:08,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:08,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:08,516 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,516 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,517 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:08,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:08,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:08,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:08,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:08,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:08,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2da40ebe[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:08,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:08,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,524 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:08,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:08,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:08,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:08,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:08,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:08,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:08,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:08,528 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:08,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:08,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:08,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:08,528 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:08,530 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,530 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,530 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,534 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:08,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:08,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:08,534 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:08,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,534 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:08,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:08,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229428555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:08,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229428559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/47d5ee15a3ab41798d49bd9cebb41f09 2023-04-23 05:56:08,598 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/19dff9dabf75402eba9897ca05e5c5ba as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/19dff9dabf75402eba9897ca05e5c5ba 2023-04-23 05:56:08,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/19dff9dabf75402eba9897ca05e5c5ba, entries=250, sequenceid=627, filesize=16.7 K 2023-04-23 05:56:08,602 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/e3d36d3f901747fc947c0b3bac432d10 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/e3d36d3f901747fc947c0b3bac432d10 2023-04-23 05:56:08,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/e3d36d3f901747fc947c0b3bac432d10, entries=150, sequenceid=627, filesize=11.9 K 2023-04-23 05:56:08,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/47d5ee15a3ab41798d49bd9cebb41f09 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/47d5ee15a3ab41798d49bd9cebb41f09 2023-04-23 05:56:08,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/47d5ee15a3ab41798d49bd9cebb41f09, entries=150, sequenceid=627, filesize=11.9 K 2023-04-23 05:56:08,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 70eda440525723289f2daf7f6251db0d in 1250ms, sequenceid=627, compaction requested=true 2023-04-23 05:56:08,610 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:08,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:08,610 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:56:08,610 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:08,610 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:56:08,611 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:08,612 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66696 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:56:08,612 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:56:08,612 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,612 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/2896ed3ccef44ac29428a734410e65e9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/750850e28ce24cc584cac134feea48b6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/343ad97320434559a3b837607a231cb5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e4b2159aa978434eb2d6591e63092ddf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/19dff9dabf75402eba9897ca05e5c5ba] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=65.1 K 2023-04-23 05:56:08,612 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:56:08,612 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:56:08,612 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 2896ed3ccef44ac29428a734410e65e9, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1682229365670 2023-04-23 05:56:08,612 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,613 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/3bc7cf5cb9944319b5ba0cfe9b5df332, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/8e65c3e2da5941039f070b1276a91347, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f1d7ab62d51d4ea2b449a3610079bd96, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c935dd35aedc41268ee014fd3844a75e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/e3d36d3f901747fc947c0b3bac432d10] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=60.4 K 2023-04-23 05:56:08,613 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 750850e28ce24cc584cac134feea48b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682229365809 2023-04-23 05:56:08,613 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3bc7cf5cb9944319b5ba0cfe9b5df332, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1682229365670 2023-04-23 05:56:08,613 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 343ad97320434559a3b837607a231cb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1682229366145 2023-04-23 05:56:08,613 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 8e65c3e2da5941039f070b1276a91347, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682229365809 2023-04-23 05:56:08,614 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e4b2159aa978434eb2d6591e63092ddf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1682229367300 2023-04-23 05:56:08,614 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f1d7ab62d51d4ea2b449a3610079bd96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1682229366145 2023-04-23 05:56:08,614 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 19dff9dabf75402eba9897ca05e5c5ba, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682229367333 2023-04-23 05:56:08,614 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c935dd35aedc41268ee014fd3844a75e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1682229367300 2023-04-23 05:56:08,620 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e3d36d3f901747fc947c0b3bac432d10, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682229367333 2023-04-23 05:56:08,628 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#481 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:08,630 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#482 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:08,635 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:08,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:08,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:08,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:08,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:08,644 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:08,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:08,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:08,646 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,646 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,648 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:08,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:08,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:08,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:08,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:08,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:08,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:08,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:08,650 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,650 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:08,650 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-23 05:56:08,651 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:08,651 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:08,651 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:08,651 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:08,651 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:08,651 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:08,661 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/01f7a4472b7e46aca902b2b056f31215 2023-04-23 05:56:08,676 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/1f9cd01fe31e4e42a7b8f9ddf15007e0 2023-04-23 05:56:08,686 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/13d802b5d22545afb18ce99577b42644 2023-04-23 05:56:08,690 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/01f7a4472b7e46aca902b2b056f31215 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/01f7a4472b7e46aca902b2b056f31215 2023-04-23 05:56:08,693 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/01f7a4472b7e46aca902b2b056f31215, entries=150, sequenceid=640, filesize=11.9 K 2023-04-23 05:56:08,694 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/1f9cd01fe31e4e42a7b8f9ddf15007e0 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1f9cd01fe31e4e42a7b8f9ddf15007e0 2023-04-23 05:56:08,697 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1f9cd01fe31e4e42a7b8f9ddf15007e0, entries=150, sequenceid=640, filesize=11.9 K 2023-04-23 05:56:08,698 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/13d802b5d22545afb18ce99577b42644 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/13d802b5d22545afb18ce99577b42644 2023-04-23 05:56:08,703 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/13d802b5d22545afb18ce99577b42644, entries=150, sequenceid=640, filesize=11.9 K 2023-04-23 05:56:08,704 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 70eda440525723289f2daf7f6251db0d in 54ms, sequenceid=640, compaction requested=true 2023-04-23 05:56:08,704 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:08,704 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:08,704 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:08,708 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,709 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,709 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,709 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:08,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b24be25[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:08,712 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:08,713 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,714 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:08,714 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:08,714 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,714 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:08,719 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:08,719 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,719 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,719 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:08,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:08,723 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:08,723 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:08,723 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:08,723 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:08,723 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:08,725 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,726 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,726 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,726 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,727 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,729 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,729 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,730 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:08,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:08,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:08,730 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,731 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:08,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,731 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:08,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,831 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:08,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:08,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:08,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:08,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:08,834 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:08,834 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,835 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:08,835 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:08,835 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,835 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,836 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,837 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,837 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,837 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:08,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:08,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:08,839 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,839 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:08,839 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:08,839 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:08,841 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,841 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,842 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,842 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:08,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b683de[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:08,844 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:08,845 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,845 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,846 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:08,846 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:08,846 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:08,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,846 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:08,847 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,847 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:08,847 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,847 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:08,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:08,849 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:08,849 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:08,849 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:08,849 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:08,849 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:08,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,851 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,851 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,852 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,853 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,856 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,856 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,856 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,857 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,856 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:08,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:08,864 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:08,864 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:08,971 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:08,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:08,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:08,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:08,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:08,975 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:08,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:08,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:08,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,975 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,976 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,977 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:08,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:08,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:08,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:08,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:08,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:08,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:08,978 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:08,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:08,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:08,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,979 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:08,980 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:08,981 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,981 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:08,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:08,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a8ddd4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:08,983 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:08,983 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,985 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:08,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:08,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,985 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:08,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:08,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:08,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:08,989 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:08,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:08,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:08,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:08,990 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:08,992 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,992 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:08,992 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:08,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,993 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,994 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:08,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:08,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:08,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:08,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:08,996 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,997 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:08,997 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:08,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:08,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:08,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:08,997 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:08,997 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:08,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,038 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/ec7834f211964838ab1be3a738107573 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/ec7834f211964838ab1be3a738107573 2023-04-23 05:56:09,042 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into ec7834f211964838ab1be3a738107573(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:09,042 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:09,042 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=11, startTime=1682229368610; duration=0sec 2023-04-23 05:56:09,042 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:09,042 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:56:09,044 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74002 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:56:09,044 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/C is initiating minor compaction (all files) 2023-04-23 05:56:09,044 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/C in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,044 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/591648664b0649518239675d5c4274dc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/db41f9333e7a4e78817a73b72fae4b66, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/254e1c22b8044bbeb4a0505b0b94f500, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/06c1c1eef4b84c188a4dd7452985fe5b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/47d5ee15a3ab41798d49bd9cebb41f09, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/13d802b5d22545afb18ce99577b42644] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=72.3 K 2023-04-23 05:56:09,044 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 591648664b0649518239675d5c4274dc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1682229365670 2023-04-23 05:56:09,045 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting db41f9333e7a4e78817a73b72fae4b66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1682229365809 2023-04-23 05:56:09,045 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 254e1c22b8044bbeb4a0505b0b94f500, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1682229366145 2023-04-23 05:56:09,046 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 06c1c1eef4b84c188a4dd7452985fe5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1682229367300 2023-04-23 05:56:09,046 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 47d5ee15a3ab41798d49bd9cebb41f09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682229367333 2023-04-23 05:56:09,046 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 13d802b5d22545afb18ce99577b42644, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229367389 2023-04-23 05:56:09,054 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/42fa49a26f044e10af1413d339216d65 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/42fa49a26f044e10af1413d339216d65 2023-04-23 05:56:09,058 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#C#compaction#486 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:09,062 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into 42fa49a26f044e10af1413d339216d65(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:09,062 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:09,062 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=11, startTime=1682229368610; duration=0sec 2023-04-23 05:56:09,062 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:09,062 WARN [DataStreamer for file /user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/636ae7a02b3047788d9e23a31446c36a] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-23 05:56:09,066 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/636ae7a02b3047788d9e23a31446c36a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/636ae7a02b3047788d9e23a31446c36a 2023-04-23 05:56:09,074 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 70eda440525723289f2daf7f6251db0d/C of 70eda440525723289f2daf7f6251db0d into 636ae7a02b3047788d9e23a31446c36a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:09,074 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:09,074 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/C, priority=10, startTime=1682229368611; duration=0sec 2023-04-23 05:56:09,074 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:09,097 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:09,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:09,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:09,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:09,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:09,101 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:09,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:09,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:09,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,101 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,102 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,102 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,103 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,103 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:09,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:09,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:09,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:09,104 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,104 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:09,104 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:09,104 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:09,104 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:09,106 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,106 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,107 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,107 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:09,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c74f8a1[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:09,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:09,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:09,110 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:09,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:09,111 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:09,112 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:09,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:09,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:09,115 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:09,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:09,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:09,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:09,115 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:09,117 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,117 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,117 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,119 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,124 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:09,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:09,124 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,125 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:09,125 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:09,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,225 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:09,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:09,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:09,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:09,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:09,228 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:09,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:09,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:09,229 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,229 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,230 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:09,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:09,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:09,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:09,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:09,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:09,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:09,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:09,232 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:09,234 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,234 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,234 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,235 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:09,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:09,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22dfb09c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:09,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:09,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:09,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:09,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,238 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:09,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:09,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:09,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:09,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:09,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:09,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:09,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:09,246 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,246 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,246 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,248 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,250 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:09,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:09,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:09,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,250 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:09,250 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:09,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,250 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,350 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:09,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:09,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:09,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:09,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:09,354 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:09,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:09,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:09,354 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,356 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:09,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:09,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:09,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:09,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:09,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:09,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:09,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:09,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:09,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:09,358 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:09,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:09,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77226f6a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:09,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:09,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,364 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,364 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:09,364 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:09,364 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:09,364 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,364 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:09,365 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,365 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:09,368 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:09,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:09,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:09,368 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:09,368 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:09,369 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,369 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,369 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:09,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:09,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:09,379 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,379 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,380 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:09,380 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:09,380 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,480 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:09,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:09,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:09,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:09,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:09,487 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:09,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:09,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:09,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,488 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,489 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,489 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,489 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,490 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:09,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:09,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:09,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:09,491 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:09,491 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,491 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:09,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:09,492 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:09,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:09,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:09,492 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:09,493 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,494 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:09,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60735a27[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:09,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:09,496 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:09,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:09,497 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:09,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:09,500 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:09,503 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:09,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:09,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:09,503 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:09,503 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:09,505 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,505 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,505 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,509 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,513 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:09,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:09,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,514 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:09,514 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:09,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:09,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:56:09,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:09,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:09,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:09,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:09,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:09,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:09,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/cedd5732510549a6a6b9ab01c9be5108 2023-04-23 05:56:09,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/22615e3b5be3411ea592d23761a2bdc1 2023-04-23 05:56:09,614 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:09,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:09,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:09,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:09,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:09,618 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:09,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:09,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:09,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,620 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,620 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:09,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:09,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:09,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:09,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:09,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:09,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:09,622 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:09,622 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,622 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:09,623 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:09,623 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,623 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:09,623 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:09,623 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,623 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:09,623 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:09,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,624 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:09,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b391778[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:09,627 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:09,627 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,628 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:09,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:09,628 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:09,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:09,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229369604, max timestamp=1682229369628Num uniques 859; 2023-04-23 05:56:09,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229369604, max timestamp=1682229369628Num uniques 859; 2023-04-23 05:56:09,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:09,630 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229369604, max timestamp=1682229369628Num uniques 859; 2023-04-23 05:56:09,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:09,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:09,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:09,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,631 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:09,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:09,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:09,633 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:09,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:09,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:09,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:09,633 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:09,635 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,635 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,635 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,637 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,638 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:09,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:09,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,641 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:09,641 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:09,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:09,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:09,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229429682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:09,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229429687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,741 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:09,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:09,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:09,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:09,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:09,744 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:09,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:09,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:09,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,746 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:09,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:09,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:09,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:09,747 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:09,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:09,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:09,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:09,748 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:09,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:09,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:09,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:09,750 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:09,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:09,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44bd0fd3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:09,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:09,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,754 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:09,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:09,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:09,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,755 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:09,755 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:09,755 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,755 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:09,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:09,758 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:09,758 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:09,758 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:09,758 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:09,758 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:09,760 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,760 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,760 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,761 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,762 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:09,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:09,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,766 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:09,766 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:09,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:09,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229429789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:09,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229429790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,866 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:09,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:09,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:09,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:09,870 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:09,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:09,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:09,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,873 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:09,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:09,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:09,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:09,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,874 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:09,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:09,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:09,876 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,876 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:09,876 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:09,876 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:09,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:09,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:09,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:09,876 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:09,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,877 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,878 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:09,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:09,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20806111[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:09,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:09,879 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,880 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:09,880 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:09,881 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:09,881 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,881 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:09,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,882 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:09,882 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,882 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:09,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:09,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:09,884 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:09,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:09,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:09,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:09,884 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:09,887 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,887 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,887 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:09,888 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:09,889 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:09,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:09,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:09,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:09,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:09,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:09,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:09,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:09,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:09,891 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:09,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,892 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:09,892 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:09,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:09,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:09,992 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:09,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:09,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:09,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:09,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:09,995 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:09,995 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:09,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:09,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:09,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:09,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:10,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:10,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:10,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:10,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:10,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:10,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:10,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:10,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:10,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:10,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:10,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:10,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:10,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:10,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:10,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:10,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:10,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:10,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:10,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:10,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:10,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229429993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:10,003 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:10,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229429995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:10,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:10,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:10,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:10,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad8a544[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:10,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:10,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:10,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:10,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:10,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:10,010 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:10,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:10,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:10,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:10,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:10,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:10,012 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:10,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:10,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:10,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:10,012 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:10,012 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/22416a674fff4e069a46d68575beb7e9 2023-04-23 05:56:10,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:10,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,014 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:10,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:10,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:10,016 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:10,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:10,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/cedd5732510549a6a6b9ab01c9be5108 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cedd5732510549a6a6b9ab01c9be5108 2023-04-23 05:56:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:10,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:10,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cedd5732510549a6a6b9ab01c9be5108, entries=200, sequenceid=659, filesize=14.3 K 2023-04-23 05:56:10,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/22615e3b5be3411ea592d23761a2bdc1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/22615e3b5be3411ea592d23761a2bdc1 2023-04-23 05:56:10,021 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,021 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:10,021 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:10,021 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,021 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:10,021 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:10,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:10,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:10,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:10,022 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:10,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:10,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:10,022 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:10,022 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:10,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:10,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:10,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/22615e3b5be3411ea592d23761a2bdc1, entries=150, sequenceid=659, filesize=11.9 K 2023-04-23 05:56:10,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/22416a674fff4e069a46d68575beb7e9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/22416a674fff4e069a46d68575beb7e9 2023-04-23 05:56:10,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/22416a674fff4e069a46d68575beb7e9, entries=150, sequenceid=659, filesize=11.9 K 2023-04-23 05:56:10,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 70eda440525723289f2daf7f6251db0d in 452ms, sequenceid=659, compaction requested=true 2023-04-23 05:56:10,029 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:10,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:10,030 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:10,030 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:10,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:10,030 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:10,030 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:10,031 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:10,031 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/A is initiating minor compaction (all files) 2023-04-23 05:56:10,031 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 70eda440525723289f2daf7f6251db0d/B is initiating minor compaction (all files) 2023-04-23 05:56:10,031 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/A in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:10,031 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70eda440525723289f2daf7f6251db0d/B in TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:10,031 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/ec7834f211964838ab1be3a738107573, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/01f7a4472b7e46aca902b2b056f31215, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cedd5732510549a6a6b9ab01c9be5108] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=39.1 K 2023-04-23 05:56:10,031 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/42fa49a26f044e10af1413d339216d65, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1f9cd01fe31e4e42a7b8f9ddf15007e0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/22615e3b5be3411ea592d23761a2bdc1] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp, totalSize=36.7 K 2023-04-23 05:56:10,031 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting ec7834f211964838ab1be3a738107573, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682229367333 2023-04-23 05:56:10,031 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 42fa49a26f044e10af1413d339216d65, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682229367333 2023-04-23 05:56:10,031 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 01f7a4472b7e46aca902b2b056f31215, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229367389 2023-04-23 05:56:10,032 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1f9cd01fe31e4e42a7b8f9ddf15007e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229367389 2023-04-23 05:56:10,032 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting cedd5732510549a6a6b9ab01c9be5108, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682229369573 2023-04-23 05:56:10,032 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 22615e3b5be3411ea592d23761a2bdc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1682229369573 2023-04-23 05:56:10,038 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#A#compaction#490 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:10,040 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 70eda440525723289f2daf7f6251db0d#B#compaction#491 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:10,122 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:10,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:10,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:10,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:10,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:10,126 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:10,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:10,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:10,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:10,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:10,127 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:10,127 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:10,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:10,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:10,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:10,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:10,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:10,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:10,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:10,129 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:10,129 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:10,129 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:10,130 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:10,130 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:10,130 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:10,130 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:10,130 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-23 05:56:10,130 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:10,130 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:10,130 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:10,130 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:10,130 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:10,130 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:10,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:10,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:10,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229430328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229430333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229430435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229430439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,449 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/5cde1b41c2ea44d8b7dccbfd66fefcd4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/5cde1b41c2ea44d8b7dccbfd66fefcd4 2023-04-23 05:56:10,463 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/7bbba9deb1a74bb9a0a5c880ce0faed7 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/7bbba9deb1a74bb9a0a5c880ce0faed7 2023-04-23 05:56:10,466 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/B of 70eda440525723289f2daf7f6251db0d into 5cde1b41c2ea44d8b7dccbfd66fefcd4(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:10,466 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:10,466 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/B, priority=13, startTime=1682229370030; duration=0sec 2023-04-23 05:56:10,466 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:10,466 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-23 05:56:10,467 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-23 05:56:10,467 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-23 05:56:10,467 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. because compaction request was cancelled 2023-04-23 05:56:10,468 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70eda440525723289f2daf7f6251db0d/A of 70eda440525723289f2daf7f6251db0d into 7bbba9deb1a74bb9a0a5c880ce0faed7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:10,468 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:10,468 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d., storeName=70eda440525723289f2daf7f6251db0d/A, priority=13, startTime=1682229370030; duration=0sec 2023-04-23 05:56:10,468 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:10,539 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/34f88965c0a048ca96245aeb78fb74d1 2023-04-23 05:56:10,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229430640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229430651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38344 deadline: 1682229430943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:10,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:38368 deadline: 1682229430954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,968 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/c208ea81399d4c759c513abfe5e23c97 2023-04-23 05:56:10,978 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=695 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/55bf85fdd5de49b1baa132642f6722c9 2023-04-23 05:56:10,983 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/34f88965c0a048ca96245aeb78fb74d1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/34f88965c0a048ca96245aeb78fb74d1 2023-04-23 05:56:10,986 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/34f88965c0a048ca96245aeb78fb74d1, entries=150, sequenceid=695, filesize=11.9 K 2023-04-23 05:56:10,987 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/c208ea81399d4c759c513abfe5e23c97 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c208ea81399d4c759c513abfe5e23c97 2023-04-23 05:56:10,990 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c208ea81399d4c759c513abfe5e23c97, entries=150, sequenceid=695, filesize=11.9 K 2023-04-23 05:56:10,991 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/55bf85fdd5de49b1baa132642f6722c9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/55bf85fdd5de49b1baa132642f6722c9 2023-04-23 05:56:10,994 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/55bf85fdd5de49b1baa132642f6722c9, entries=150, sequenceid=695, filesize=11.9 K 2023-04-23 05:56:10,995 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 70eda440525723289f2daf7f6251db0d in 865ms, sequenceid=695, compaction requested=true 2023-04-23 05:56:10,995 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:10,995 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:10,995 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:10,995 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:10,995 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:10,996 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:10,997 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,997 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:10,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:10,998 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:11,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@909ebb0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:11,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:11,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:11,001 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:11,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:11,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:11,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:11,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:11,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:11,006 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:11,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:11,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:11,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:11,006 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:11,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:11,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,010 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:11,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:11,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:11,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:11,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:11,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:11,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:11,013 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:11,013 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:11,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:11,113 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:11,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:11,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:11,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:11,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:11,116 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:11,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:11,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:11,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:11,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,116 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,117 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:11,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:11,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:11,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:11,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:11,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:11,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:11,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:11,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:11,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:11,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:11,119 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:11,119 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:11,119 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:56:11,119 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:11,119 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:11,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:11,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:11,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:11,120 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:11,132 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=707 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/53bd4f4d0e9d44d5be3f8c6a61715cbc 2023-04-23 05:56:11,146 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=707 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/1de063f85f82453dbb0798c5bae9b7ac 2023-04-23 05:56:11,157 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=707 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1c72b1a4bdcc4a8a99fad91810183a34 2023-04-23 05:56:11,165 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/53bd4f4d0e9d44d5be3f8c6a61715cbc as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/53bd4f4d0e9d44d5be3f8c6a61715cbc 2023-04-23 05:56:11,170 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/53bd4f4d0e9d44d5be3f8c6a61715cbc, entries=100, sequenceid=707, filesize=9.5 K 2023-04-23 05:56:11,171 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/1de063f85f82453dbb0798c5bae9b7ac as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1de063f85f82453dbb0798c5bae9b7ac 2023-04-23 05:56:11,175 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1de063f85f82453dbb0798c5bae9b7ac, entries=100, sequenceid=707, filesize=9.5 K 2023-04-23 05:56:11,176 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1c72b1a4bdcc4a8a99fad91810183a34 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c72b1a4bdcc4a8a99fad91810183a34 2023-04-23 05:56:11,191 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c72b1a4bdcc4a8a99fad91810183a34, entries=100, sequenceid=707, filesize=9.5 K 2023-04-23 05:56:11,192 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 70eda440525723289f2daf7f6251db0d in 73ms, sequenceid=707, compaction requested=true 2023-04-23 05:56:11,192 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:11,192 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:11,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:11,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:11,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:11,192 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,196 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:11,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2566aed5[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:11,197 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:11,198 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,198 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,199 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,199 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:11,199 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:11,199 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:11,199 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:11,200 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:11,200 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:11,200 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:11,200 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:11,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:11,203 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:11,203 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:11,203 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:11,203 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:11,203 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:11,204 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,204 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:11,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,205 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,205 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:11,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:11,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:11,209 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:11,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:11,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:11,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,209 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,209 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:11,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:11,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:11,209 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:11,310 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:11,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:11,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:11,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:11,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:11,317 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:11,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:11,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:11,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:11,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,318 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:11,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:11,319 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,319 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:11,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:11,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:11,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:11,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:11,324 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:11,324 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:11,324 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:11,324 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:11,326 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,327 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,327 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cdcb81[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:11,328 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:11,328 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,333 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,333 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:11,333 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:11,333 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:11,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:11,334 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:11,334 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:11,335 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:11,336 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:11,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:11,338 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:11,339 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:11,339 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:11,339 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:11,339 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:11,340 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,340 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:11,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:11,340 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,342 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:11,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:11,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,345 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:11,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:11,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:11,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:11,345 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:11,346 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:11,346 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:11,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:11,446 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:11,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:11,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:11,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:11,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:11,449 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:11,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:11,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:11,449 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:11,450 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,450 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,452 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,452 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:11,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:11,452 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,452 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:11,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:11,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:11,454 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:11,454 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:11,454 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:11,454 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:11,454 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:11,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:11,454 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. started... 2023-04-23 05:56:11,455 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-23 05:56:11,456 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:11,456 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:11,456 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:11,457 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:11,457 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:11,457 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:11,465 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/83370d105a7c4c078a63218b44fe097e 2023-04-23 05:56:11,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:11,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. as already flushing 2023-04-23 05:56:11,479 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/58c3a337792b45229c712498e00356e6 2023-04-23 05:56:11,490 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/9ddfe58bec264473b370dde8303567f8 2023-04-23 05:56:11,494 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/83370d105a7c4c078a63218b44fe097e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/83370d105a7c4c078a63218b44fe097e 2023-04-23 05:56:11,495 DEBUG [Thread-3048] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0adb6988 to 127.0.0.1:63802 2023-04-23 05:56:11,495 DEBUG [Thread-3048] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:11,495 DEBUG [Thread-3037] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14f9adf3 to 127.0.0.1:63802 2023-04-23 05:56:11,495 DEBUG [Thread-3037] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:11,495 DEBUG [Thread-3042] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36d2d826 to 127.0.0.1:63802 2023-04-23 05:56:11,495 DEBUG [Thread-3042] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:11,497 DEBUG [Thread-3029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32cd672d to 127.0.0.1:63802 2023-04-23 05:56:11,497 DEBUG [Thread-3029] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:11,498 DEBUG [Thread-3040] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78fa3dee to 127.0.0.1:63802 2023-04-23 05:56:11,498 DEBUG [Thread-3040] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:11,500 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/83370d105a7c4c078a63218b44fe097e, entries=50, sequenceid=712, filesize=7.1 K 2023-04-23 05:56:11,501 DEBUG [Thread-3046] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ece3076 to 127.0.0.1:63802 2023-04-23 05:56:11,501 DEBUG [Thread-3046] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:11,502 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/58c3a337792b45229c712498e00356e6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/58c3a337792b45229c712498e00356e6 2023-04-23 05:56:11,505 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/58c3a337792b45229c712498e00356e6, entries=50, sequenceid=712, filesize=7.1 K 2023-04-23 05:56:11,505 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/9ddfe58bec264473b370dde8303567f8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9ddfe58bec264473b370dde8303567f8 2023-04-23 05:56:11,507 DEBUG [Thread-3044] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6871d7d8 to 127.0.0.1:63802 2023-04-23 05:56:11,507 DEBUG [Thread-3044] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:11,508 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9ddfe58bec264473b370dde8303567f8, entries=50, sequenceid=712, filesize=7.1 K 2023-04-23 05:56:11,508 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=93.93 KB/96180 for 70eda440525723289f2daf7f6251db0d in 54ms, sequenceid=712, compaction requested=true 2023-04-23 05:56:11,508 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:11,508 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:11,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:11,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:11,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:11,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:11,510 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,510 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1920bc99[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:11,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:11,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,513 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:11,513 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:11,513 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,513 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:11,513 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:11,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:11,514 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:11,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,515 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:11,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:11,517 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:11,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:11,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:11,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:11,517 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:11,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,518 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:11,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:11,518 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:11,519 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:11,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:11,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:11,521 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:11,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:11,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:11,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:11,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:11,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:11,521 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:11,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:11,521 DEBUG [Thread-3039] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:11,522 DEBUG [Thread-3039] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:11,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:11,622 DEBUG [Thread-3039] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:11,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:14,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:14,911 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-23 05:56:14,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:14,911 DEBUG [Thread-3031] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09b8ff30 to 127.0.0.1:63802 2023-04-23 05:56:14,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:14,911 DEBUG [Thread-3031] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:14,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:14,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:14,911 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:14,911 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:14,947 DEBUG [Thread-3035] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28e3107b to 127.0.0.1:63802 2023-04-23 05:56:14,947 DEBUG [Thread-3035] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:14,968 DEBUG [Thread-3033] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c050db8 to 127.0.0.1:63802 2023-04-23 05:56:14,968 DEBUG [Thread-3033] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 236 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 34 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 274 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 2102 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 6306 rows 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 2117 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 6348 rows 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 2091 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 6273 rows 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 2108 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 6320 rows 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(402): scanned 2110 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(403): verified 6330 rows 2023-04-23 05:56:14,969 INFO [Listener at localhost/41845] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-23 05:56:14,970 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4539ed0c to 127.0.0.1:63802 2023-04-23 05:56:14,970 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:14,971 INFO [Listener at localhost/41845] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-23 05:56:14,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-23 05:56:14,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:14,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-23 05:56:14,976 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229374976"}]},"ts":"1682229374976"} 2023-04-23 05:56:14,977 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-23 05:56:14,980 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-23 05:56:14,980 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=70eda440525723289f2daf7f6251db0d, UNASSIGN}] 2023-04-23 05:56:14,982 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=70eda440525723289f2daf7f6251db0d, UNASSIGN 2023-04-23 05:56:14,982 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=70eda440525723289f2daf7f6251db0d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:14,982 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229374982"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229374982"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229374982"}]},"ts":"1682229374982"} 2023-04-23 05:56:14,983 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:56:15,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-23 05:56:15,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:15,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 70eda440525723289f2daf7f6251db0d, disabling compactions & flushes 2023-04-23 05:56:15,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:15,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-23 05:56:15,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/5a587d164500422b80dcae39ed28dd58 2023-04-23 05:56:15,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-23 05:56:15,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/61090b646cba4a31b56fb5d0479e54d9 2023-04-23 05:56:16,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-23 05:56:16,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1ca116663d90483fa23632d51f0f98ab 2023-04-23 05:56:16,139 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/5a587d164500422b80dcae39ed28dd58 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5a587d164500422b80dcae39ed28dd58 2023-04-23 05:56:16,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5a587d164500422b80dcae39ed28dd58, entries=150, sequenceid=730, filesize=11.9 K 2023-04-23 05:56:16,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/61090b646cba4a31b56fb5d0479e54d9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/61090b646cba4a31b56fb5d0479e54d9 2023-04-23 05:56:16,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/61090b646cba4a31b56fb5d0479e54d9, entries=150, sequenceid=730, filesize=11.9 K 2023-04-23 05:56:16,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/1ca116663d90483fa23632d51f0f98ab as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1ca116663d90483fa23632d51f0f98ab 2023-04-23 05:56:16,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1ca116663d90483fa23632d51f0f98ab, entries=150, sequenceid=730, filesize=11.9 K 2023-04-23 05:56:16,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=13.42 KB/13740 for 70eda440525723289f2daf7f6251db0d in 1239ms, sequenceid=730, compaction requested=true 2023-04-23 05:56:16,150 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:16,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:16,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:16,150 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. because compaction request was cancelled 2023-04-23 05:56:16,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:16,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:16,151 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. because compaction request was cancelled 2023-04-23 05:56:16,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. after waiting 0 ms 2023-04-23 05:56:16,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:16,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:16,151 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 70eda440525723289f2daf7f6251db0d 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-23 05:56:16,151 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. because compaction request was cancelled 2023-04-23 05:56:16,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=A 2023-04-23 05:56:16,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:16,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=B 2023-04-23 05:56:16,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:16,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70eda440525723289f2daf7f6251db0d, store=C 2023-04-23 05:56:16,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:16,557 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/88d048df61a649d884e637bebde670e2 2023-04-23 05:56:16,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/d736970f17a443d9a8492b1ff9921a21 2023-04-23 05:56:17,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-23 05:56:17,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=735 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/ad9cb76b18764cc39e470145ae6ec9d4 2023-04-23 05:56:17,376 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/A/88d048df61a649d884e637bebde670e2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/88d048df61a649d884e637bebde670e2 2023-04-23 05:56:17,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/88d048df61a649d884e637bebde670e2, entries=100, sequenceid=735, filesize=9.5 K 2023-04-23 05:56:17,380 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/B/d736970f17a443d9a8492b1ff9921a21 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/d736970f17a443d9a8492b1ff9921a21 2023-04-23 05:56:17,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/d736970f17a443d9a8492b1ff9921a21, entries=100, sequenceid=735, filesize=9.5 K 2023-04-23 05:56:17,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/.tmp/C/ad9cb76b18764cc39e470145ae6ec9d4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ad9cb76b18764cc39e470145ae6ec9d4 2023-04-23 05:56:17,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ad9cb76b18764cc39e470145ae6ec9d4, entries=100, sequenceid=735, filesize=9.5 K 2023-04-23 05:56:17,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 70eda440525723289f2daf7f6251db0d in 1236ms, sequenceid=735, compaction requested=true 2023-04-23 05:56:17,388 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/270f0fbf9abc4917aa02425f9e2cffda, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1a316184305845fcb12ba20c4e50e120, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/4c28ed8a0e9f4049846908e5a9eaad7b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a7cc7d2a72974a1d92533eb1e25d9e10, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a80c45db56c04da8a4497f603536427a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/6386884ee76a42e5809a9c93fcf5ad7d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/33e3edc5cc5d4b5c8aebaa304f0950f5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f58e8da79bc141839f80a04d5a5f5c40, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a1879bec24d6483cb9a4e8dbd8f7eacf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/38dffb138eef4dbd8d3e40194f9236ae, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9a3a528ae72f4dd2947cd151dabbea89, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/0a73a6c0b5574a7d840f1571e51f78ff, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cf9c2e5bfcd4402590184be122e9db8f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9ae92832623148a8bd1f1f3b1116fe48, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/14557c4db40145ada919d4c94050af03, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/b4a69546892f4d9e8ffed6956633b766, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5938e222fa314da382d3c3f96ee992aa, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dfb2da64b5a0452faddd6d9b563b8d72, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/35e43ea67b4a46b283e4bd20d0f0cf2c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/c21124dccb304b9baec101d952a4fd19, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/3650dabe0d3c4448b6a5138cf4f48be7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1c7acc7df28f4b40bddfd65cd9712133, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dd1efa6fdd334062a88f86e68d50ee0b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e1630bbc8a3248048d140304d14c80d4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f89ba522e3cf4f4281c73bfd33715414, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/fa9a5513a7a74d4f9306b8a72dd40852, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/2896ed3ccef44ac29428a734410e65e9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/750850e28ce24cc584cac134feea48b6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/343ad97320434559a3b837607a231cb5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e4b2159aa978434eb2d6591e63092ddf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/19dff9dabf75402eba9897ca05e5c5ba, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/ec7834f211964838ab1be3a738107573, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/01f7a4472b7e46aca902b2b056f31215, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cedd5732510549a6a6b9ab01c9be5108] to archive 2023-04-23 05:56:17,389 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:56:17,391 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/270f0fbf9abc4917aa02425f9e2cffda to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/270f0fbf9abc4917aa02425f9e2cffda 2023-04-23 05:56:17,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1a316184305845fcb12ba20c4e50e120 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1a316184305845fcb12ba20c4e50e120 2023-04-23 05:56:17,393 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/4c28ed8a0e9f4049846908e5a9eaad7b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/4c28ed8a0e9f4049846908e5a9eaad7b 2023-04-23 05:56:17,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a7cc7d2a72974a1d92533eb1e25d9e10 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a7cc7d2a72974a1d92533eb1e25d9e10 2023-04-23 05:56:17,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a80c45db56c04da8a4497f603536427a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a80c45db56c04da8a4497f603536427a 2023-04-23 05:56:17,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/6386884ee76a42e5809a9c93fcf5ad7d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/6386884ee76a42e5809a9c93fcf5ad7d 2023-04-23 05:56:17,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/33e3edc5cc5d4b5c8aebaa304f0950f5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/33e3edc5cc5d4b5c8aebaa304f0950f5 2023-04-23 05:56:17,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f58e8da79bc141839f80a04d5a5f5c40 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f58e8da79bc141839f80a04d5a5f5c40 2023-04-23 05:56:17,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a1879bec24d6483cb9a4e8dbd8f7eacf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/a1879bec24d6483cb9a4e8dbd8f7eacf 2023-04-23 05:56:17,400 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/38dffb138eef4dbd8d3e40194f9236ae to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/38dffb138eef4dbd8d3e40194f9236ae 2023-04-23 05:56:17,401 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9a3a528ae72f4dd2947cd151dabbea89 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9a3a528ae72f4dd2947cd151dabbea89 2023-04-23 05:56:17,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/0a73a6c0b5574a7d840f1571e51f78ff to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/0a73a6c0b5574a7d840f1571e51f78ff 2023-04-23 05:56:17,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cf9c2e5bfcd4402590184be122e9db8f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cf9c2e5bfcd4402590184be122e9db8f 2023-04-23 05:56:17,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9ae92832623148a8bd1f1f3b1116fe48 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/9ae92832623148a8bd1f1f3b1116fe48 2023-04-23 05:56:17,405 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/14557c4db40145ada919d4c94050af03 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/14557c4db40145ada919d4c94050af03 2023-04-23 05:56:17,406 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/b4a69546892f4d9e8ffed6956633b766 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/b4a69546892f4d9e8ffed6956633b766 2023-04-23 05:56:17,407 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5938e222fa314da382d3c3f96ee992aa to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5938e222fa314da382d3c3f96ee992aa 2023-04-23 05:56:17,408 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dfb2da64b5a0452faddd6d9b563b8d72 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dfb2da64b5a0452faddd6d9b563b8d72 2023-04-23 05:56:17,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/35e43ea67b4a46b283e4bd20d0f0cf2c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/35e43ea67b4a46b283e4bd20d0f0cf2c 2023-04-23 05:56:17,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/c21124dccb304b9baec101d952a4fd19 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/c21124dccb304b9baec101d952a4fd19 2023-04-23 05:56:17,410 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/3650dabe0d3c4448b6a5138cf4f48be7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/3650dabe0d3c4448b6a5138cf4f48be7 2023-04-23 05:56:17,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1c7acc7df28f4b40bddfd65cd9712133 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/1c7acc7df28f4b40bddfd65cd9712133 2023-04-23 05:56:17,412 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dd1efa6fdd334062a88f86e68d50ee0b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/dd1efa6fdd334062a88f86e68d50ee0b 2023-04-23 05:56:17,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e1630bbc8a3248048d140304d14c80d4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e1630bbc8a3248048d140304d14c80d4 2023-04-23 05:56:17,415 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f89ba522e3cf4f4281c73bfd33715414 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/f89ba522e3cf4f4281c73bfd33715414 2023-04-23 05:56:17,416 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/fa9a5513a7a74d4f9306b8a72dd40852 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/fa9a5513a7a74d4f9306b8a72dd40852 2023-04-23 05:56:17,417 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/2896ed3ccef44ac29428a734410e65e9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/2896ed3ccef44ac29428a734410e65e9 2023-04-23 05:56:17,417 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/750850e28ce24cc584cac134feea48b6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/750850e28ce24cc584cac134feea48b6 2023-04-23 05:56:17,418 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/343ad97320434559a3b837607a231cb5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/343ad97320434559a3b837607a231cb5 2023-04-23 05:56:17,419 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e4b2159aa978434eb2d6591e63092ddf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/e4b2159aa978434eb2d6591e63092ddf 2023-04-23 05:56:17,420 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/19dff9dabf75402eba9897ca05e5c5ba to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/19dff9dabf75402eba9897ca05e5c5ba 2023-04-23 05:56:17,421 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/ec7834f211964838ab1be3a738107573 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/ec7834f211964838ab1be3a738107573 2023-04-23 05:56:17,422 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/01f7a4472b7e46aca902b2b056f31215 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/01f7a4472b7e46aca902b2b056f31215 2023-04-23 05:56:17,423 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cedd5732510549a6a6b9ab01c9be5108 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/cedd5732510549a6a6b9ab01c9be5108 2023-04-23 05:56:17,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/ec55f638120f4badb33279145138744e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9388010a918641c5a21c09943750bde2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1d4f0f83ae484153a4fbdf85320be02b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a3d50961a2cd4c99a382518a2f10dc4f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/5a9fd7779a264a28b232c1c0b183d614, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f00d095ce37543e3a959f7e9b24349da, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/63ebccf4ad9d41369a946b9a967d93b9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1ac624a7d69a4f25b47f4d8271bc6f23, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7a719c0f74e8493abec430ec2bbf557b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/4a4c31dc70c14e9ebc9cf9eb2d199560, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/831494818ea04158a00b67b4b72c8513, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/be06530a39424970aebf66789ce18ab8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c7a93e6df8d64dc2b7c0bd13007dee14, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/68232384752649468a74816c9e668102, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/b6ce4f6aecbd4806b96e5ee5766ae09f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e8af13676714adfa3c4534063817056, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1879b28da4d840bf84695e6027c35473, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9decc9a7bc0c4f11912a36126a3ab20e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7af14f5582d84155b52a5ecee51de320, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/88620571ddfe4e919e10a614f7067c57, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/dc942ae2b1c8498d98bc5c816fc1a178, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/0d1dca50b67f405d8d9070a693213bc3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f57bd9a049e043dcaecf60fc927bad80, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e7d41c04d8541aca4da3d61265a379d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a030038c4eca44debfb18d9576dc354d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/3bc7cf5cb9944319b5ba0cfe9b5df332, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f83814602ba04ccba6fc72ad510b250c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/8e65c3e2da5941039f070b1276a91347, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f1d7ab62d51d4ea2b449a3610079bd96, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c935dd35aedc41268ee014fd3844a75e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/42fa49a26f044e10af1413d339216d65, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/e3d36d3f901747fc947c0b3bac432d10, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1f9cd01fe31e4e42a7b8f9ddf15007e0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/22615e3b5be3411ea592d23761a2bdc1] to archive 2023-04-23 05:56:17,432 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:56:17,433 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/ec55f638120f4badb33279145138744e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/ec55f638120f4badb33279145138744e 2023-04-23 05:56:17,434 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9388010a918641c5a21c09943750bde2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9388010a918641c5a21c09943750bde2 2023-04-23 05:56:17,434 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1d4f0f83ae484153a4fbdf85320be02b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1d4f0f83ae484153a4fbdf85320be02b 2023-04-23 05:56:17,435 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a3d50961a2cd4c99a382518a2f10dc4f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a3d50961a2cd4c99a382518a2f10dc4f 2023-04-23 05:56:17,436 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/5a9fd7779a264a28b232c1c0b183d614 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/5a9fd7779a264a28b232c1c0b183d614 2023-04-23 05:56:17,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f00d095ce37543e3a959f7e9b24349da to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f00d095ce37543e3a959f7e9b24349da 2023-04-23 05:56:17,439 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/63ebccf4ad9d41369a946b9a967d93b9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/63ebccf4ad9d41369a946b9a967d93b9 2023-04-23 05:56:17,440 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1ac624a7d69a4f25b47f4d8271bc6f23 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1ac624a7d69a4f25b47f4d8271bc6f23 2023-04-23 05:56:17,441 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7a719c0f74e8493abec430ec2bbf557b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7a719c0f74e8493abec430ec2bbf557b 2023-04-23 05:56:17,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/4a4c31dc70c14e9ebc9cf9eb2d199560 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/4a4c31dc70c14e9ebc9cf9eb2d199560 2023-04-23 05:56:17,443 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/831494818ea04158a00b67b4b72c8513 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/831494818ea04158a00b67b4b72c8513 2023-04-23 05:56:17,444 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/be06530a39424970aebf66789ce18ab8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/be06530a39424970aebf66789ce18ab8 2023-04-23 05:56:17,444 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c7a93e6df8d64dc2b7c0bd13007dee14 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c7a93e6df8d64dc2b7c0bd13007dee14 2023-04-23 05:56:17,445 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/68232384752649468a74816c9e668102 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/68232384752649468a74816c9e668102 2023-04-23 05:56:17,446 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/b6ce4f6aecbd4806b96e5ee5766ae09f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/b6ce4f6aecbd4806b96e5ee5766ae09f 2023-04-23 05:56:17,447 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e8af13676714adfa3c4534063817056 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e8af13676714adfa3c4534063817056 2023-04-23 05:56:17,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1879b28da4d840bf84695e6027c35473 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1879b28da4d840bf84695e6027c35473 2023-04-23 05:56:17,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9decc9a7bc0c4f11912a36126a3ab20e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/9decc9a7bc0c4f11912a36126a3ab20e 2023-04-23 05:56:17,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7af14f5582d84155b52a5ecee51de320 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/7af14f5582d84155b52a5ecee51de320 2023-04-23 05:56:17,450 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/88620571ddfe4e919e10a614f7067c57 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/88620571ddfe4e919e10a614f7067c57 2023-04-23 05:56:17,451 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/dc942ae2b1c8498d98bc5c816fc1a178 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/dc942ae2b1c8498d98bc5c816fc1a178 2023-04-23 05:56:17,451 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/0d1dca50b67f405d8d9070a693213bc3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/0d1dca50b67f405d8d9070a693213bc3 2023-04-23 05:56:17,452 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f57bd9a049e043dcaecf60fc927bad80 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f57bd9a049e043dcaecf60fc927bad80 2023-04-23 05:56:17,453 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e7d41c04d8541aca4da3d61265a379d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/6e7d41c04d8541aca4da3d61265a379d 2023-04-23 05:56:17,454 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a030038c4eca44debfb18d9576dc354d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/a030038c4eca44debfb18d9576dc354d 2023-04-23 05:56:17,455 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/3bc7cf5cb9944319b5ba0cfe9b5df332 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/3bc7cf5cb9944319b5ba0cfe9b5df332 2023-04-23 05:56:17,455 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f83814602ba04ccba6fc72ad510b250c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f83814602ba04ccba6fc72ad510b250c 2023-04-23 05:56:17,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/8e65c3e2da5941039f070b1276a91347 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/8e65c3e2da5941039f070b1276a91347 2023-04-23 05:56:17,457 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f1d7ab62d51d4ea2b449a3610079bd96 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/f1d7ab62d51d4ea2b449a3610079bd96 2023-04-23 05:56:17,458 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c935dd35aedc41268ee014fd3844a75e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c935dd35aedc41268ee014fd3844a75e 2023-04-23 05:56:17,458 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/42fa49a26f044e10af1413d339216d65 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/42fa49a26f044e10af1413d339216d65 2023-04-23 05:56:17,459 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/e3d36d3f901747fc947c0b3bac432d10 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/e3d36d3f901747fc947c0b3bac432d10 2023-04-23 05:56:17,460 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1f9cd01fe31e4e42a7b8f9ddf15007e0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1f9cd01fe31e4e42a7b8f9ddf15007e0 2023-04-23 05:56:17,461 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/22615e3b5be3411ea592d23761a2bdc1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/22615e3b5be3411ea592d23761a2bdc1 2023-04-23 05:56:17,463 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5aa3c012815944cc910b01b446a502b7, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/19947a82b7034b7491ebc79fb936ce5b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/e34cbbba0ff7498da8086e175184416c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ffccb791853d471da705c8b26549ed5f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c59f418dadf466a822241390037cc24, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/3f18bf9f6a38412b8e740688deabf2d9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/915805e477824db58d147aff06b1afd0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9fdbc0ae304c4e0a9e9a866e4d6e7c9c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/97dd85be66cd4e9090d1757a75f94851, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5d1ea8c6398c454d9feb2d0d610ff4db, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/84d6f0dcea084cfb9b7364ce679cacee, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/7f70ecea01f04cc6abcf53a2f47429fa, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2e31d35cb1134ea39d1aa9fdf1e4d620, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b2449c246209427e9f01cef66722a8a2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d5add1b081e04375868504bd83e55b45, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/f80b9669fa7042539a6e6f05a4bb713e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1f4745474d724f7886382245dcced426, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1eb1021e5bce4101b82534845feb0e67, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c69aa29aea204143a32e9da29b1039bf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b4ceef151a2f481f9e7986d531bff6a1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/6b193562598d4653aa28e0a6d31ac3c8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/78dc3ccd5a2c470c86afe967129d1664, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2237bfee41974e6ea3d419a7755db652, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/58d18a164549447fbce89d46a8fdd505, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c27bc7c7d27246c7b82b6c5aff87ea17, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/591648664b0649518239675d5c4274dc, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d1dc23237718441f88ba61a3487af088, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/db41f9333e7a4e78817a73b72fae4b66, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/254e1c22b8044bbeb4a0505b0b94f500, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/06c1c1eef4b84c188a4dd7452985fe5b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/47d5ee15a3ab41798d49bd9cebb41f09, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/13d802b5d22545afb18ce99577b42644] to archive 2023-04-23 05:56:17,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:56:17,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5aa3c012815944cc910b01b446a502b7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5aa3c012815944cc910b01b446a502b7 2023-04-23 05:56:17,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/19947a82b7034b7491ebc79fb936ce5b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/19947a82b7034b7491ebc79fb936ce5b 2023-04-23 05:56:17,467 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/e34cbbba0ff7498da8086e175184416c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/e34cbbba0ff7498da8086e175184416c 2023-04-23 05:56:17,468 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ffccb791853d471da705c8b26549ed5f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ffccb791853d471da705c8b26549ed5f 2023-04-23 05:56:17,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c59f418dadf466a822241390037cc24 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c59f418dadf466a822241390037cc24 2023-04-23 05:56:17,470 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/3f18bf9f6a38412b8e740688deabf2d9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/3f18bf9f6a38412b8e740688deabf2d9 2023-04-23 05:56:17,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/915805e477824db58d147aff06b1afd0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/915805e477824db58d147aff06b1afd0 2023-04-23 05:56:17,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9fdbc0ae304c4e0a9e9a866e4d6e7c9c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9fdbc0ae304c4e0a9e9a866e4d6e7c9c 2023-04-23 05:56:17,472 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/97dd85be66cd4e9090d1757a75f94851 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/97dd85be66cd4e9090d1757a75f94851 2023-04-23 05:56:17,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5d1ea8c6398c454d9feb2d0d610ff4db to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/5d1ea8c6398c454d9feb2d0d610ff4db 2023-04-23 05:56:17,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/84d6f0dcea084cfb9b7364ce679cacee to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/84d6f0dcea084cfb9b7364ce679cacee 2023-04-23 05:56:17,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/7f70ecea01f04cc6abcf53a2f47429fa to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/7f70ecea01f04cc6abcf53a2f47429fa 2023-04-23 05:56:17,475 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2e31d35cb1134ea39d1aa9fdf1e4d620 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2e31d35cb1134ea39d1aa9fdf1e4d620 2023-04-23 05:56:17,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b2449c246209427e9f01cef66722a8a2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b2449c246209427e9f01cef66722a8a2 2023-04-23 05:56:17,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d5add1b081e04375868504bd83e55b45 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d5add1b081e04375868504bd83e55b45 2023-04-23 05:56:17,477 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/f80b9669fa7042539a6e6f05a4bb713e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/f80b9669fa7042539a6e6f05a4bb713e 2023-04-23 05:56:17,478 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1f4745474d724f7886382245dcced426 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1f4745474d724f7886382245dcced426 2023-04-23 05:56:17,478 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1eb1021e5bce4101b82534845feb0e67 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1eb1021e5bce4101b82534845feb0e67 2023-04-23 05:56:17,479 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c69aa29aea204143a32e9da29b1039bf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c69aa29aea204143a32e9da29b1039bf 2023-04-23 05:56:17,480 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b4ceef151a2f481f9e7986d531bff6a1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/b4ceef151a2f481f9e7986d531bff6a1 2023-04-23 05:56:17,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/6b193562598d4653aa28e0a6d31ac3c8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/6b193562598d4653aa28e0a6d31ac3c8 2023-04-23 05:56:17,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/78dc3ccd5a2c470c86afe967129d1664 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/78dc3ccd5a2c470c86afe967129d1664 2023-04-23 05:56:17,482 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2237bfee41974e6ea3d419a7755db652 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/2237bfee41974e6ea3d419a7755db652 2023-04-23 05:56:17,483 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/58d18a164549447fbce89d46a8fdd505 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/58d18a164549447fbce89d46a8fdd505 2023-04-23 05:56:17,483 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c27bc7c7d27246c7b82b6c5aff87ea17 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/c27bc7c7d27246c7b82b6c5aff87ea17 2023-04-23 05:56:17,484 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/591648664b0649518239675d5c4274dc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/591648664b0649518239675d5c4274dc 2023-04-23 05:56:17,485 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d1dc23237718441f88ba61a3487af088 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/d1dc23237718441f88ba61a3487af088 2023-04-23 05:56:17,486 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/db41f9333e7a4e78817a73b72fae4b66 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/db41f9333e7a4e78817a73b72fae4b66 2023-04-23 05:56:17,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/254e1c22b8044bbeb4a0505b0b94f500 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/254e1c22b8044bbeb4a0505b0b94f500 2023-04-23 05:56:17,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/06c1c1eef4b84c188a4dd7452985fe5b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/06c1c1eef4b84c188a4dd7452985fe5b 2023-04-23 05:56:17,488 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/47d5ee15a3ab41798d49bd9cebb41f09 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/47d5ee15a3ab41798d49bd9cebb41f09 2023-04-23 05:56:17,489 DEBUG [StoreCloser-TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/13d802b5d22545afb18ce99577b42644 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/13d802b5d22545afb18ce99577b42644 2023-04-23 05:56:17,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/recovered.edits/738.seqid, newMaxSeqId=738, maxSeqId=1 2023-04-23 05:56:17,495 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d. 2023-04-23 05:56:17,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 70eda440525723289f2daf7f6251db0d: 2023-04-23 05:56:17,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:17,496 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=70eda440525723289f2daf7f6251db0d, regionState=CLOSED 2023-04-23 05:56:17,496 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229377496"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229377496"}]},"ts":"1682229377496"} 2023-04-23 05:56:17,498 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-23 05:56:17,498 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 70eda440525723289f2daf7f6251db0d, server=jenkins-hbase4.apache.org,40995,1682229223026 in 2.5140 sec 2023-04-23 05:56:17,499 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-23 05:56:17,500 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=70eda440525723289f2daf7f6251db0d, UNASSIGN in 2.5180 sec 2023-04-23 05:56:17,500 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229377500"}]},"ts":"1682229377500"} 2023-04-23 05:56:17,501 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-23 05:56:17,502 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-23 05:56:17,503 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.5310 sec 2023-04-23 05:56:19,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-23 05:56:19,082 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-23 05:56:19,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-23 05:56:19,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:19,085 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:19,086 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:19,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-23 05:56:19,089 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:19,091 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/recovered.edits] 2023-04-23 05:56:19,094 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/34f88965c0a048ca96245aeb78fb74d1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/34f88965c0a048ca96245aeb78fb74d1 2023-04-23 05:56:19,095 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/53bd4f4d0e9d44d5be3f8c6a61715cbc to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/53bd4f4d0e9d44d5be3f8c6a61715cbc 2023-04-23 05:56:19,096 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5a587d164500422b80dcae39ed28dd58 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/5a587d164500422b80dcae39ed28dd58 2023-04-23 05:56:19,098 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/7bbba9deb1a74bb9a0a5c880ce0faed7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/7bbba9deb1a74bb9a0a5c880ce0faed7 2023-04-23 05:56:19,099 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/83370d105a7c4c078a63218b44fe097e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/83370d105a7c4c078a63218b44fe097e 2023-04-23 05:56:19,100 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/88d048df61a649d884e637bebde670e2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/A/88d048df61a649d884e637bebde670e2 2023-04-23 05:56:19,102 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1de063f85f82453dbb0798c5bae9b7ac to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/1de063f85f82453dbb0798c5bae9b7ac 2023-04-23 05:56:19,103 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/58c3a337792b45229c712498e00356e6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/58c3a337792b45229c712498e00356e6 2023-04-23 05:56:19,104 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/5cde1b41c2ea44d8b7dccbfd66fefcd4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/5cde1b41c2ea44d8b7dccbfd66fefcd4 2023-04-23 05:56:19,106 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/61090b646cba4a31b56fb5d0479e54d9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/61090b646cba4a31b56fb5d0479e54d9 2023-04-23 05:56:19,107 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c208ea81399d4c759c513abfe5e23c97 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/c208ea81399d4c759c513abfe5e23c97 2023-04-23 05:56:19,108 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/d736970f17a443d9a8492b1ff9921a21 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/B/d736970f17a443d9a8492b1ff9921a21 2023-04-23 05:56:19,110 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c72b1a4bdcc4a8a99fad91810183a34 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1c72b1a4bdcc4a8a99fad91810183a34 2023-04-23 05:56:19,111 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1ca116663d90483fa23632d51f0f98ab to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/1ca116663d90483fa23632d51f0f98ab 2023-04-23 05:56:19,112 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/22416a674fff4e069a46d68575beb7e9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/22416a674fff4e069a46d68575beb7e9 2023-04-23 05:56:19,113 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/55bf85fdd5de49b1baa132642f6722c9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/55bf85fdd5de49b1baa132642f6722c9 2023-04-23 05:56:19,114 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/636ae7a02b3047788d9e23a31446c36a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/636ae7a02b3047788d9e23a31446c36a 2023-04-23 05:56:19,115 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9ddfe58bec264473b370dde8303567f8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/9ddfe58bec264473b370dde8303567f8 2023-04-23 05:56:19,115 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ad9cb76b18764cc39e470145ae6ec9d4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/C/ad9cb76b18764cc39e470145ae6ec9d4 2023-04-23 05:56:19,117 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/recovered.edits/738.seqid to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d/recovered.edits/738.seqid 2023-04-23 05:56:19,118 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/70eda440525723289f2daf7f6251db0d 2023-04-23 05:56:19,118 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:56:19,120 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:19,121 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-23 05:56:19,122 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-23 05:56:19,123 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:19,123 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-23 05:56:19,123 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682229379123"}]},"ts":"9223372036854775807"} 2023-04-23 05:56:19,125 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-23 05:56:19,125 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 70eda440525723289f2daf7f6251db0d, NAME => 'TestAcidGuarantees,,1682229349304.70eda440525723289f2daf7f6251db0d.', STARTKEY => '', ENDKEY => ''}] 2023-04-23 05:56:19,125 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-23 05:56:19,125 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682229379125"}]},"ts":"9223372036854775807"} 2023-04-23 05:56:19,126 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-23 05:56:19,128 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:19,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 46 msec 2023-04-23 05:56:19,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-23 05:56:19,187 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-23 05:56:19,198 INFO [Listener at localhost/41845] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 238), OpenFileDescriptor=562 (was 569), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=405 (was 384) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=12659 (was 12662) 2023-04-23 05:56:19,210 INFO [Listener at localhost/41845] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=562, MaxFileDescriptor=60000, SystemLoadAverage=405, ProcessCount=169, AvailableMemoryMB=12659 2023-04-23 05:56:19,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-23 05:56:19,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:56:19,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:19,215 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-23 05:56:19,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-23 05:56:19,215 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-23 05:56:19,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-23 05:56:19,216 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:19,217 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48 empty. 2023-04-23 05:56:19,217 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:19,217 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:56:19,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-23 05:56:19,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-23 05:56:19,624 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-23 05:56:19,625 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9ad1ed5a4f4248a6e5f0e43c7f776f48, NAME => 'TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp 2023-04-23 05:56:19,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-23 05:56:20,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:56:20,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9ad1ed5a4f4248a6e5f0e43c7f776f48, disabling compactions & flushes 2023-04-23 05:56:20,031 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:20,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:20,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. after waiting 0 ms 2023-04-23 05:56:20,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:20,031 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:20,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:20,033 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-23 05:56:20,034 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229380034"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229380034"}]},"ts":"1682229380034"} 2023-04-23 05:56:20,035 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-23 05:56:20,035 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-23 05:56:20,035 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229380035"}]},"ts":"1682229380035"} 2023-04-23 05:56:20,036 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-23 05:56:20,040 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, ASSIGN}] 2023-04-23 05:56:20,041 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, ASSIGN 2023-04-23 05:56:20,042 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=false 2023-04-23 05:56:20,193 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=9ad1ed5a4f4248a6e5f0e43c7f776f48, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:20,193 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229380193"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229380193"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229380193"}]},"ts":"1682229380193"} 2023-04-23 05:56:20,195 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:56:20,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-23 05:56:20,351 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:20,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9ad1ed5a4f4248a6e5f0e43c7f776f48, NAME => 'TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:56:20,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:20,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:56:20,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:20,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:20,353 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:20,354 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A 2023-04-23 05:56:20,354 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A 2023-04-23 05:56:20,354 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:56:20,355 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9ad1ed5a4f4248a6e5f0e43c7f776f48 columnFamilyName A 2023-04-23 05:56:20,355 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(310): Store=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:56:20,355 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:20,356 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B 2023-04-23 05:56:20,356 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B 2023-04-23 05:56:20,356 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:56:20,357 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9ad1ed5a4f4248a6e5f0e43c7f776f48 columnFamilyName B 2023-04-23 05:56:20,357 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(310): Store=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:56:20,357 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:20,358 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C 2023-04-23 05:56:20,358 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C 2023-04-23 05:56:20,358 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:56:20,359 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9ad1ed5a4f4248a6e5f0e43c7f776f48 columnFamilyName C 2023-04-23 05:56:20,359 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(310): Store=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:56:20,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:20,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:20,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:20,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:56:20,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:20,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-23 05:56:20,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9ad1ed5a4f4248a6e5f0e43c7f776f48; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65550086, jitterRate=-0.023227602243423462}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:56:20,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:20,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., pid=56, masterSystemTime=1682229380346 2023-04-23 05:56:20,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:20,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:20,367 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=9ad1ed5a4f4248a6e5f0e43c7f776f48, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:20,367 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229380367"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229380367"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229380367"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229380367"}]},"ts":"1682229380367"} 2023-04-23 05:56:20,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-23 05:56:20,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 in 173 msec 2023-04-23 05:56:20,370 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-23 05:56:20,370 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, ASSIGN in 329 msec 2023-04-23 05:56:20,370 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-23 05:56:20,371 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229380371"}]},"ts":"1682229380371"} 2023-04-23 05:56:20,371 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-23 05:56:20,373 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-23 05:56:20,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-23 05:56:21,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-23 05:56:21,320 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-23 05:56:21,321 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x52104714 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,326 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9d61148, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,327 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,328 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55860, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,329 DEBUG [Listener at localhost/41845] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-23 05:56:21,332 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-23 05:56:21,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-23 05:56:21,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-23 05:56:21,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:21,344 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-23 05:56:21,345 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-23 05:56:21,345 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-23 05:56:21,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-23 05:56:21,348 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, REOPEN/MOVE}] 2023-04-23 05:56:21,349 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, REOPEN/MOVE 2023-04-23 05:56:21,349 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=9ad1ed5a4f4248a6e5f0e43c7f776f48, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:21,350 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229381349"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229381349"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229381349"}]},"ts":"1682229381349"} 2023-04-23 05:56:21,350 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:56:21,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9ad1ed5a4f4248a6e5f0e43c7f776f48, disabling compactions & flushes 2023-04-23 05:56:21,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:21,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:21,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. after waiting 0 ms 2023-04-23 05:56:21,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:21,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-23 05:56:21,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:21,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:21,509 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 9ad1ed5a4f4248a6e5f0e43c7f776f48 to self. 2023-04-23 05:56:21,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,510 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=9ad1ed5a4f4248a6e5f0e43c7f776f48, regionState=CLOSED 2023-04-23 05:56:21,511 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229381510"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229381510"}]},"ts":"1682229381510"} 2023-04-23 05:56:21,513 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-23 05:56:21,513 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 in 162 msec 2023-04-23 05:56:21,513 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,40995,1682229223026; forceNewPlan=false, retain=true 2023-04-23 05:56:21,664 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=9ad1ed5a4f4248a6e5f0e43c7f776f48, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:21,664 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229381664"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229381664"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229381664"}]},"ts":"1682229381664"} 2023-04-23 05:56:21,665 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:56:21,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:21,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9ad1ed5a4f4248a6e5f0e43c7f776f48, NAME => 'TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.', STARTKEY => '', ENDKEY => ''} 2023-04-23 05:56:21,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-23 05:56:21,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,821 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,822 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A 2023-04-23 05:56:21,822 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A 2023-04-23 05:56:21,822 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:56:21,823 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9ad1ed5a4f4248a6e5f0e43c7f776f48 columnFamilyName A 2023-04-23 05:56:21,825 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(310): Store=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:56:21,825 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,826 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B 2023-04-23 05:56:21,826 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B 2023-04-23 05:56:21,826 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:56:21,826 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9ad1ed5a4f4248a6e5f0e43c7f776f48 columnFamilyName B 2023-04-23 05:56:21,826 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(310): Store=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:56:21,826 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,827 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C 2023-04-23 05:56:21,827 DEBUG [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C 2023-04-23 05:56:21,827 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-23 05:56:21,827 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9ad1ed5a4f4248a6e5f0e43c7f776f48 columnFamilyName C 2023-04-23 05:56:21,828 INFO [StoreOpener-9ad1ed5a4f4248a6e5f0e43c7f776f48-1] regionserver.HStore(310): Store=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-23 05:56:21,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:21,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-23 05:56:21,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9ad1ed5a4f4248a6e5f0e43c7f776f48; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67368443, jitterRate=0.0038680285215377808}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-23 05:56:21,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:21,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., pid=61, masterSystemTime=1682229381816 2023-04-23 05:56:21,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:21,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:21,834 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=9ad1ed5a4f4248a6e5f0e43c7f776f48, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:21,835 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229381834"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682229381834"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682229381834"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682229381834"}]},"ts":"1682229381834"} 2023-04-23 05:56:21,836 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-23 05:56:21,836 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 in 170 msec 2023-04-23 05:56:21,837 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-23 05:56:21,837 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, REOPEN/MOVE in 488 msec 2023-04-23 05:56:21,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-23 05:56:21,839 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 492 msec 2023-04-23 05:56:21,840 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 505 msec 2023-04-23 05:56:21,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-23 05:56:21,841 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x4a320817 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,846 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a4acea2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,846 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x19c1e3ee to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,850 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43693271, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,850 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x526f2221 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,857 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44277511, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,857 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x17cf2e3f to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,861 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c1a68cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,861 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x3420e54c to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,865 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a709b21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,866 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x156e4ba6 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,872 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b1973b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,873 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x785e366b to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,880 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f0a71bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,881 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x5fdfef3b to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,888 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56bb77c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,888 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x77f9399b to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,891 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@577fa668, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,892 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(139): Connect 0x430757a6 to 127.0.0.1:63802 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-23 05:56:21,898 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d8c8bbc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-23 05:56:21,905 DEBUG [hconnection-0x2783829f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,905 DEBUG [hconnection-0x3cbc7e1f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:21,908 DEBUG [hconnection-0x2c43c88c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,909 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55894, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,914 DEBUG [hconnection-0x4284c99b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,914 DEBUG [hconnection-0x39730a44-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,914 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55868, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,914 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55882, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,915 DEBUG [hconnection-0x30ea88e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,914 DEBUG [hconnection-0x7dbb5ea9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,924 DEBUG [hconnection-0x104b5c29-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,914 DEBUG [hconnection-0x411cbefc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,915 DEBUG [hconnection-0x54de77b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-23 05:56:21,926 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55900, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,926 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55906, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:21,928 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55916, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:21,929 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:21,929 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:21,930 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:21,930 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55948, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,930 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:21,930 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55936, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:21,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:21,934 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55950, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:21,935 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:21,934 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55932, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-23 05:56:21,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:21,935 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:21,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:21,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:21,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:21,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:21,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:21,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:21,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:21,940 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:21,940 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:21,940 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:21,940 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:21,941 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-23 05:56:21,970 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:21,970 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:21,970 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:21,970 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:21,970 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:21,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:21,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:21,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:21,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682229381978, max timestamp=1682229381984Num uniques 859; 2023-04-23 05:56:21,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:21,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682229381978, max timestamp=1682229381984Num uniques 859; 2023-04-23 05:56:21,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:21,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682229381978, max timestamp=1682229381984Num uniques 859; 2023-04-23 05:56:21,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:21,996 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d4ac67cac6134873bc94b10d88402b39 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d4ac67cac6134873bc94b10d88402b39 2023-04-23 05:56:22,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229441998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,001 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e132fc7418c341a19941f700eb2a1aa0 2023-04-23 05:56:22,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229441999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229441998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229442000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229442002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,031 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/e48e414f6c274416817ca2d081198e55 2023-04-23 05:56:22,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229442103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229442103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229442105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229442105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229442105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229442305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229442306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229442308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229442308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229442309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,447 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/9f2cc029d0074f3aa671d0cfd0a27695 2023-04-23 05:56:22,452 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e132fc7418c341a19941f700eb2a1aa0 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e132fc7418c341a19941f700eb2a1aa0 2023-04-23 05:56:22,456 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e132fc7418c341a19941f700eb2a1aa0, entries=150, sequenceid=13, filesize=25.1 K 2023-04-23 05:56:22,456 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/e48e414f6c274416817ca2d081198e55 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e48e414f6c274416817ca2d081198e55 2023-04-23 05:56:22,460 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e48e414f6c274416817ca2d081198e55, entries=150, sequenceid=13, filesize=11.6 K 2023-04-23 05:56:22,461 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/9f2cc029d0074f3aa671d0cfd0a27695 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9f2cc029d0074f3aa671d0cfd0a27695 2023-04-23 05:56:22,464 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9f2cc029d0074f3aa671d0cfd0a27695, entries=150, sequenceid=13, filesize=11.6 K 2023-04-23 05:56:22,465 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 524ms, sequenceid=13, compaction requested=false 2023-04-23 05:56:22,465 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:22,465 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:22,465 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:22,470 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,470 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:22,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fb28066[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:22,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:22,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,473 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,473 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,473 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:22,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,473 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:22,473 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:22,473 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:22,476 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,476 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:22,476 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:22,476 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:22,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:22,491 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:22,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:22,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:22,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:22,491 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:22,492 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,492 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,492 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:22,493 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,494 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:22,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:22,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:22,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:22,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:22,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:22,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,504 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:22,504 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:22,604 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:22,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:22,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:22,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:22,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:22,608 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:22,608 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:22,608 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:22,608 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:22,608 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,609 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:22,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:22,610 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:22,611 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,611 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:22,611 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-23 05:56:22,611 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:22,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-23 05:56:22,611 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:22,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-23 05:56:22,611 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:22,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-23 05:56:22,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:22,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:22,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:22,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:22,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:22,616 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:22,616 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:22,616 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:22,616 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:22,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:22,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229442616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229442616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229442616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229442617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229442618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:22,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:22,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb28d43[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:22,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:22,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,624 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:22,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:22,624 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:22,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:22,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,625 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:22,630 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:22,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:22,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:22,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:22,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:22,631 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:22,631 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423a3e5e17975f14fcfab448adb1ba30989 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a3e5e17975f14fcfab448adb1ba30989 2023-04-23 05:56:22,633 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,633 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:22,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,633 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,634 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,637 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/666ed33d75b14d6f9cdc31e651597981 2023-04-23 05:56:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:22,640 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:22,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:22,641 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:22,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:22,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:22,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:22,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:22,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:22,641 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:22,641 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:22,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,641 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,642 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,649 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/f4fd204e81954f1f88dd62d509898bd3 2023-04-23 05:56:22,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229442720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229442720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229442721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229442721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229442721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,741 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:22,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:22,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:22,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:22,745 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:22,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:22,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:22,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:22,745 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,747 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:22,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:22,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:22,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:22,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:22,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:22,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:22,748 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:22,748 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:22,748 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:22,749 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:22,749 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:22,749 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:22,750 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,750 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:22,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@274bc6f7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:22,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:22,752 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,753 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:22,754 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,754 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:22,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:22,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:22,756 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:22,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,756 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:22,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:22,759 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:22,759 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:22,759 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:22,759 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:22,759 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:22,760 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,760 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,760 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:22,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,761 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,762 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:22,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:22,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:22,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:22,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:22,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:22,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:22,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:22,764 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,765 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:22,765 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:22,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:22,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:22,865 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:22,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:22,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:22,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:22,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:22,868 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:22,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:22,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:22,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:22,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,869 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,870 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:22,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:22,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,870 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:22,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:22,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:22,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:22,871 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:22,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:22,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:22,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:22,872 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:22,872 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:22,874 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,875 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:22,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b9362f3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:22,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:22,876 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,883 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,883 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:22,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:22,883 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:22,883 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:22,884 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:22,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:22,887 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:22,887 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:22,887 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:22,887 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:22,887 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:22,890 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,890 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,890 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:22,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:22,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,894 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:22,894 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:22,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:22,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:22,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:22,894 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:22,894 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,895 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,895 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:22,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:22,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:22,895 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,895 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,895 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:22,895 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,895 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:22,895 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:22,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229442924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229442924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229442924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229442924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:22,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229442927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:22,995 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:22,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:22,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:22,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:22,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:22,999 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:22,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:22,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:22,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:22,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:22,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:23,000 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:23,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:23,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:23,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:23,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:23,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:23,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:23,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:23,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:23,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:23,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70697457[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:23,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:23,006 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:23,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:23,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:23,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:23,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:23,008 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:23,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,008 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:23,009 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:23,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:23,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:23,012 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:23,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:23,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:23,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:23,012 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:23,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,013 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:23,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:23,014 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:23,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,018 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:23,018 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:23,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:23,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:23,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:23,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:23,018 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:23,018 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,018 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:23,019 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:23,019 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,019 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:23,019 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:23,019 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:23,019 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:23,019 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:23,026 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-23 05:56:23,059 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/ac5a59a919d14acf9e12fe33d1184e64 2023-04-23 05:56:23,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/666ed33d75b14d6f9cdc31e651597981 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/666ed33d75b14d6f9cdc31e651597981 2023-04-23 05:56:23,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/666ed33d75b14d6f9cdc31e651597981, entries=150, sequenceid=51, filesize=25.1 K 2023-04-23 05:56:23,071 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/f4fd204e81954f1f88dd62d509898bd3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f4fd204e81954f1f88dd62d509898bd3 2023-04-23 05:56:23,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f4fd204e81954f1f88dd62d509898bd3, entries=150, sequenceid=51, filesize=11.6 K 2023-04-23 05:56:23,076 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/ac5a59a919d14acf9e12fe33d1184e64 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac5a59a919d14acf9e12fe33d1184e64 2023-04-23 05:56:23,083 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac5a59a919d14acf9e12fe33d1184e64, entries=150, sequenceid=51, filesize=11.6 K 2023-04-23 05:56:23,084 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 473ms, sequenceid=51, compaction requested=false 2023-04-23 05:56:23,084 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:23,119 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:23,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:23,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:23,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:23,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:23,132 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:23,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:23,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:23,132 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:23,133 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,133 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:23,135 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,135 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:23,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:23,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,136 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:23,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:23,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:23,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:23,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:23,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:23,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:23,138 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:23,138 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:23,138 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:23,138 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:23,138 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-23 05:56:23,138 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:23,138 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:23,139 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:23,139 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:23,139 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:23,139 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:23,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:23,231 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:23,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229383237, max timestamp=1682229383246Num uniques 859; 2023-04-23 05:56:23,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:56:23,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229383237, max timestamp=1682229383246Num uniques 859; 2023-04-23 05:56:23,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:56:23,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229383243, max timestamp=1682229383246Num uniques 859; 2023-04-23 05:56:23,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:23,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682229383252, max timestamp=1682229383270Num uniques 1118; 2023-04-23 05:56:23,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-23 05:56:23,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682229383252, max timestamp=1682229383270Num uniques 1118; 2023-04-23 05:56:23,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-23 05:56:23,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9975, totalHeapSize=22912, min timestamp=1682229383252, max timestamp=1682229383272Num uniques 1018; 2023-04-23 05:56:23,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46615, new segment data size=9975, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-23 05:56:23,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229443285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229383283, max timestamp=1682229383289Num uniques 1077; 2023-04-23 05:56:23,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-23 05:56:23,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229383283, max timestamp=1682229383289Num uniques 1077; 2023-04-23 05:56:23,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-23 05:56:23,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229383283, max timestamp=1682229383289Num uniques 1077; 2023-04-23 05:56:23,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-23 05:56:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682229383305, max timestamp=1682229383311Num uniques 1036; 2023-04-23 05:56:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682229383305, max timestamp=1682229383311Num uniques 1036; 2023-04-23 05:56:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-23 05:56:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-23 05:56:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1682229383305, max timestamp=1682229383311Num uniques 1036; 2023-04-23 05:56:23,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-23 05:56:23,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682229383324, max timestamp=1682229383331Num uniques 1095; 2023-04-23 05:56:23,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1682229383324, max timestamp=1682229383331Num uniques 1095; 2023-04-23 05:56:23,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-23 05:56:23,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682229383324, max timestamp=1682229383331Num uniques 1095; 2023-04-23 05:56:23,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-23 05:56:23,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-23 05:56:23,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229443349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229443350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229443350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229443351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229443393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229443454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229443454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229443455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229443455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,551 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304233a329da295ed4045930b0dd7e2ddaa15 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233a329da295ed4045930b0dd7e2ddaa15 2023-04-23 05:56:23,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229443596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229443656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229443657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229443657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229443659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229443899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,956 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=59, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/0f73c4baabac4c96a53909ccd13f612b 2023-04-23 05:56:23,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229443959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229443959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229443960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:23,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229443962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:23,966 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/cd8c8e0871144720b98b118efb30325c 2023-04-23 05:56:23,975 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/54101c3ecdf14aaa9fc86d7bf588bbb5 2023-04-23 05:56:23,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/0f73c4baabac4c96a53909ccd13f612b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/0f73c4baabac4c96a53909ccd13f612b 2023-04-23 05:56:23,982 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/0f73c4baabac4c96a53909ccd13f612b, entries=100, sequenceid=59, filesize=18.4 K 2023-04-23 05:56:23,990 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/cd8c8e0871144720b98b118efb30325c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cd8c8e0871144720b98b118efb30325c 2023-04-23 05:56:23,994 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cd8c8e0871144720b98b118efb30325c, entries=100, sequenceid=59, filesize=9.3 K 2023-04-23 05:56:23,995 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/54101c3ecdf14aaa9fc86d7bf588bbb5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/54101c3ecdf14aaa9fc86d7bf588bbb5 2023-04-23 05:56:23,999 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/54101c3ecdf14aaa9fc86d7bf588bbb5, entries=100, sequenceid=59, filesize=9.3 K 2023-04-23 05:56:23,999 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 861ms, sequenceid=59, compaction requested=true 2023-04-23 05:56:23,999 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:23,999 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:23,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:23,999 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:24,000 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,000 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:24,000 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:24,001 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,001 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:24,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:24,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bea278c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:24,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:24,003 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:24,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:24,005 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:24,005 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,005 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:24,006 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:24,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:24,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:24,009 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:24,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:24,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:24,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:24,009 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:24,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,012 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:24,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:24,012 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,015 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:24,015 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:24,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:24,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:24,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:24,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:24,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:24,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,016 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:24,016 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:24,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:24,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:24,116 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:24,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:24,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:24,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:24,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:24,120 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:24,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:24,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:24,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:24,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,120 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,121 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:24,121 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:24,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:24,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:24,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:24,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:24,123 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:24,123 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:24,123 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:24,123 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:24,123 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:24,123 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:24,123 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:24,123 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=147.60 KB heapSize=412.08 KB 2023-04-23 05:56:24,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:24,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:24,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:24,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:24,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:24,124 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:24,143 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423b99365350a74413fad22a17874a4e4f3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423b99365350a74413fad22a17874a4e4f3 2023-04-23 05:56:24,149 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=164, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/074fbcaab1e440ccba815a93aac89059 2023-04-23 05:56:24,158 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/f841beaeb5f943a99ca59e612411dbcf 2023-04-23 05:56:24,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:24,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:24,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:24,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229444420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:24,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229444464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229444465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:24,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229444465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55936 deadline: 1682229444465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55900 deadline: 1682229444523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,567 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/81df9bcee09e4232aef36513ccf1cbe6 2023-04-23 05:56:24,571 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/074fbcaab1e440ccba815a93aac89059 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/074fbcaab1e440ccba815a93aac89059 2023-04-23 05:56:24,575 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/074fbcaab1e440ccba815a93aac89059, entries=150, sequenceid=164, filesize=25.3 K 2023-04-23 05:56:24,576 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/f841beaeb5f943a99ca59e612411dbcf as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f841beaeb5f943a99ca59e612411dbcf 2023-04-23 05:56:24,579 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f841beaeb5f943a99ca59e612411dbcf, entries=150, sequenceid=164, filesize=11.8 K 2023-04-23 05:56:24,580 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/81df9bcee09e4232aef36513ccf1cbe6 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/81df9bcee09e4232aef36513ccf1cbe6 2023-04-23 05:56:24,583 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/81df9bcee09e4232aef36513ccf1cbe6, entries=150, sequenceid=164, filesize=11.8 K 2023-04-23 05:56:24,584 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~412.03 KB/421920, currentSize=33.54 KB/34350 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 461ms, sequenceid=164, compaction requested=true 2023-04-23 05:56:24,584 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:24,584 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:24,584 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:24,584 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:24,585 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,585 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:24,585 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:24,586 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,586 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:24,587 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:24,587 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:24,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:24,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@526f3078[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:24,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:24,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,590 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:24,590 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,590 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:24,590 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:24,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:24,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:24,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:24,592 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:24,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:24,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:24,595 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:24,595 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:24,595 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:24,595 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:24,595 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:24,599 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,599 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:24,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,599 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:24,599 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,601 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:24,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:24,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:24,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:24,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:24,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:24,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:24,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:24,604 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:24,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,604 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:24,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:24,705 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:24,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:24,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:24,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:24,708 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:24,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:24,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:24,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:24,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,708 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:24,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:24,709 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,710 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:24,710 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:24,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:24,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:24,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:24,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:24,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:24,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:24,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:24,711 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:24,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:24,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:24,711 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-23 05:56:24,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:24,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:24,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:24,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:24,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:24,711 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:24,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:24,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:24,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229384773, max timestamp=1682229384793Num uniques 859; 2023-04-23 05:56:24,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229384773, max timestamp=1682229384793Num uniques 859; 2023-04-23 05:56:24,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:24,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:24,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229384773, max timestamp=1682229384793Num uniques 859; 2023-04-23 05:56:24,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:24,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229384853, max timestamp=1682229384869Num uniques 1018; 2023-04-23 05:56:24,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229384853, max timestamp=1682229384869Num uniques 1018; 2023-04-23 05:56:24,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:56:24,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:56:24,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682229384853, max timestamp=1682229384869Num uniques 1018; 2023-04-23 05:56:24,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-23 05:56:24,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229384933, max timestamp=1682229384947Num uniques 1027; 2023-04-23 05:56:24,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-23 05:56:24,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229384933, max timestamp=1682229384947Num uniques 1027; 2023-04-23 05:56:24,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1682229384933, max timestamp=1682229384947Num uniques 1027; 2023-04-23 05:56:24,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-23 05:56:24,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-23 05:56:25,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682229384999, max timestamp=1682229385016Num uniques 1036; 2023-04-23 05:56:25,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-23 05:56:25,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682229384999, max timestamp=1682229385016Num uniques 1036; 2023-04-23 05:56:25,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-23 05:56:25,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682229384999, max timestamp=1682229385016Num uniques 1036; 2023-04-23 05:56:25,019 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-23 05:56:25,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682229385029, max timestamp=1682229385079Num uniques 1045; 2023-04-23 05:56:25,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-23 05:56:25,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682229385029, max timestamp=1682229385079Num uniques 1045; 2023-04-23 05:56:25,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1682229385029, max timestamp=1682229385079Num uniques 1045; 2023-04-23 05:56:25,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-23 05:56:25,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-23 05:56:25,139 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304233f5522e0f86c401d8811ee4d5ceb58f3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233f5522e0f86c401d8811ee4d5ceb58f3 2023-04-23 05:56:25,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682229385138, max timestamp=1682229385152Num uniques 1054; 2023-04-23 05:56:25,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682229385138, max timestamp=1682229385152Num uniques 1054; 2023-04-23 05:56:25,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-23 05:56:25,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1682229385138, max timestamp=1682229385152Num uniques 1054; 2023-04-23 05:56:25,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-23 05:56:25,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-23 05:56:25,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682229385206, max timestamp=1682229385217Num uniques 1013; 2023-04-23 05:56:25,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682229385206, max timestamp=1682229385217Num uniques 1013; 2023-04-23 05:56:25,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1682229385206, max timestamp=1682229385217Num uniques 1013; 2023-04-23 05:56:25,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-23 05:56:25,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-23 05:56:25,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-23 05:56:25,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682229385254, max timestamp=1682229385278Num uniques 1022; 2023-04-23 05:56:25,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682229385254, max timestamp=1682229385278Num uniques 1022; 2023-04-23 05:56:25,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-23 05:56:25,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1682229385254, max timestamp=1682229385278Num uniques 1022; 2023-04-23 05:56:25,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-23 05:56:25,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-23 05:56:25,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682229385324, max timestamp=1682229385342Num uniques 1031; 2023-04-23 05:56:25,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-23 05:56:25,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682229385324, max timestamp=1682229385342Num uniques 1031; 2023-04-23 05:56:25,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-23 05:56:25,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1682229385324, max timestamp=1682229385342Num uniques 1031; 2023-04-23 05:56:25,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-23 05:56:25,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682229385393, max timestamp=1682229385408Num uniques 1040; 2023-04-23 05:56:25,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682229385393, max timestamp=1682229385408Num uniques 1040; 2023-04-23 05:56:25,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1682229385393, max timestamp=1682229385408Num uniques 1040; 2023-04-23 05:56:25,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-23 05:56:25,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-23 05:56:25,411 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-23 05:56:25,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:25,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229445474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:25,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:25,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229445474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:25,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:25,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229445476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:25,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682229385460, max timestamp=1682229385476Num uniques 1049; 2023-04-23 05:56:25,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1682229385460, max timestamp=1682229385477Num uniques 1049; 2023-04-23 05:56:25,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1682229385460, max timestamp=1682229385476Num uniques 1049; 2023-04-23 05:56:25,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-23 05:56:25,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-23 05:56:25,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-23 05:56:25,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:25,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229445529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:25,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682229385510, max timestamp=1682229385531Num uniques 1058; 2023-04-23 05:56:25,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682229385510, max timestamp=1682229385531Num uniques 1058; 2023-04-23 05:56:25,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-23 05:56:25,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1682229385510, max timestamp=1682229385531Num uniques 1058; 2023-04-23 05:56:25,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-23 05:56:25,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-23 05:56:25,547 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=172, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/c3fee99ca6544ca59713813832ee919f 2023-04-23 05:56:25,562 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/46cc8b89c9d14031bf63ae031d41db44 2023-04-23 05:56:25,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682229385596, max timestamp=1682229385611Num uniques 1017; 2023-04-23 05:56:25,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682229385596, max timestamp=1682229385611Num uniques 1017; 2023-04-23 05:56:25,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1682229385596, max timestamp=1682229385611Num uniques 1017; 2023-04-23 05:56:25,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-23 05:56:25,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-23 05:56:25,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-23 05:56:25,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682229385636, max timestamp=1682229385658Num uniques 1026; 2023-04-23 05:56:25,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-23 05:56:25,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682229385636, max timestamp=1682229385658Num uniques 1026; 2023-04-23 05:56:25,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1682229385636, max timestamp=1682229385658Num uniques 1026; 2023-04-23 05:56:25,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-23 05:56:25,661 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-23 05:56:25,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:25,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229445697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:25,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682229385691, max timestamp=1682229385699Num uniques 1085; 2023-04-23 05:56:25,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-23 05:56:25,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682229385691, max timestamp=1682229385699Num uniques 1085; 2023-04-23 05:56:25,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1682229385691, max timestamp=1682229385699Num uniques 1085; 2023-04-23 05:56:25,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-23 05:56:25,702 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=8470, suffix heap size=140808,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=185 2023-04-23 05:56:25,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682229385741, max timestamp=1682229385765Num uniques 1044; 2023-04-23 05:56:25,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682229385741, max timestamp=1682229385765Num uniques 1044; 2023-04-23 05:56:25,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-23 05:56:25,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1682229385741, max timestamp=1682229385765Num uniques 1044; 2023-04-23 05:56:25,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-23 05:56:25,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=8884, suffix heap size=136544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=194 2023-04-23 05:56:25,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682229385806, max timestamp=1682229385821Num uniques 1053; 2023-04-23 05:56:25,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-23 05:56:25,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682229385806, max timestamp=1682229385821Num uniques 1053; 2023-04-23 05:56:25,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1682229385806, max timestamp=1682229385821Num uniques 1053; 2023-04-23 05:56:25,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-23 05:56:25,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=9295, suffix heap size=137480,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=203 2023-04-23 05:56:25,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:25,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229445862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:25,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682229385842, max timestamp=1682229385861Num uniques 1062; 2023-04-23 05:56:25,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-23 05:56:25,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682229385842, max timestamp=1682229385861Num uniques 1062; 2023-04-23 05:56:25,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-23 05:56:25,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1682229385842, max timestamp=1682229385861Num uniques 1062; 2023-04-23 05:56:25,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-23 05:56:25,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682229385923, max timestamp=1682229385937Num uniques 1021; 2023-04-23 05:56:25,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682229385923, max timestamp=1682229385937Num uniques 1021; 2023-04-23 05:56:25,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-23 05:56:25,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1682229385923, max timestamp=1682229385937Num uniques 1021; 2023-04-23 05:56:25,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-23 05:56:25,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-23 05:56:25,974 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/3a163f0ec3d44a4bbd6a808f24972624 2023-04-23 05:56:25,978 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/c3fee99ca6544ca59713813832ee919f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c3fee99ca6544ca59713813832ee919f 2023-04-23 05:56:25,981 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c3fee99ca6544ca59713813832ee919f, entries=150, sequenceid=172, filesize=25.3 K 2023-04-23 05:56:25,982 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/46cc8b89c9d14031bf63ae031d41db44 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/46cc8b89c9d14031bf63ae031d41db44 2023-04-23 05:56:25,985 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/46cc8b89c9d14031bf63ae031d41db44, entries=150, sequenceid=172, filesize=11.8 K 2023-04-23 05:56:25,986 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/3a163f0ec3d44a4bbd6a808f24972624 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3a163f0ec3d44a4bbd6a808f24972624 2023-04-23 05:56:25,989 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3a163f0ec3d44a4bbd6a808f24972624, entries=150, sequenceid=172, filesize=11.8 K 2023-04-23 05:56:26,002 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=127.47 KB/130530 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 1290ms, sequenceid=172, compaction requested=true 2023-04-23 05:56:26,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:26,002 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:26,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=134.18 KB heapSize=383.86 KB 2023-04-23 05:56:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:26,002 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,002 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:26,002 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:26,002 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:26,002 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:26,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:26,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:26,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,004 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:26,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cb443c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:26,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:26,007 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,008 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:26,009 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:26,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,009 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:26,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:26,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,010 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:26,010 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:26,013 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:26,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:26,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:26,013 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:26,013 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:26,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,016 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,019 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,023 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:26,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:26,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229446020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,023 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,023 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:26,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,024 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,024 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:26,024 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:26,025 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304235ffa0e1724474de78e0ba287f2138e07 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304235ffa0e1724474de78e0ba287f2138e07 2023-04-23 05:56:26,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229446023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,032 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=518, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/1210494ebb2949d0afca4cd5ab935c28 2023-04-23 05:56:26,044 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/cc200a712c564041946e27a674c2e175 2023-04-23 05:56:26,125 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:26,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:26,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:26,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229446124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:26,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:26,128 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:26,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:26,128 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:26,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,129 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,130 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,130 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,130 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:26,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:26,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:26,131 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:26,131 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:26,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,132 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229446127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,132 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:26,132 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:26,132 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:26,132 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:26,133 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,133 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,134 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,134 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:26,135 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:26,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49bb4dfe[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:26,135 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,137 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:26,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:26,137 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,137 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:26,138 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,138 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,139 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:26,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:26,142 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:26,143 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:26,143 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:26,143 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:26,143 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:26,144 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,144 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,144 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,145 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,146 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,148 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,148 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,148 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,148 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,148 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,148 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:26,149 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:26,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:26,149 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,149 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,149 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,149 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:26,149 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:26,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,249 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:26,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:26,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:26,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:26,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:26,254 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:26,254 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,254 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:26,254 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:26,254 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,254 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,256 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,256 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,257 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,257 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:26,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:26,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:26,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:26,258 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,258 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:26,258 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:26,258 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:26,258 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:26,259 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,259 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:26,259 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:26,260 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,260 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,261 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:26,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3397c502[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:26,262 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:26,262 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,273 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,273 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:26,274 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:26,274 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:26,275 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:26,275 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,275 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:26,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:26,278 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:26,278 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:26,278 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:26,278 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:26,278 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:26,281 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,281 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,281 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,281 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,283 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,290 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,290 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,290 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,290 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,290 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:26,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:26,290 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,291 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,291 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,291 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,291 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,291 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:26,291 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:26,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229446328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229446333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,391 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:26,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:26,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:26,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:26,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:26,395 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:26,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:26,395 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:26,396 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,396 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,397 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:26,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:26,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:26,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:26,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:26,398 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:26,398 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,398 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:26,399 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,399 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:26,399 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:26,399 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:26,399 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:26,401 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:26,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1850dff8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:26,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:26,403 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,404 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:26,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:26,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:26,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:26,412 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:26,412 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,412 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:26,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:26,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:26,415 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:26,415 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:26,415 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:26,415 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:26,415 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:26,416 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,416 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,417 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,418 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,419 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,423 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,423 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,423 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:26,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:26,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:26,423 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,423 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:26,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,423 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,423 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,424 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,424 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,424 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,424 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,423 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:26,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/ac9a7c7bad184c1f8fa25eaa1967e8cd 2023-04-23 05:56:26,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/1210494ebb2949d0afca4cd5ab935c28 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1210494ebb2949d0afca4cd5ab935c28 2023-04-23 05:56:26,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1210494ebb2949d0afca4cd5ab935c28, entries=200, sequenceid=518, filesize=32.3 K 2023-04-23 05:56:26,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/cc200a712c564041946e27a674c2e175 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cc200a712c564041946e27a674c2e175 2023-04-23 05:56:26,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cc200a712c564041946e27a674c2e175, entries=150, sequenceid=518, filesize=11.9 K 2023-04-23 05:56:26,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/ac9a7c7bad184c1f8fa25eaa1967e8cd as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac9a7c7bad184c1f8fa25eaa1967e8cd 2023-04-23 05:56:26,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac9a7c7bad184c1f8fa25eaa1967e8cd, entries=150, sequenceid=518, filesize=11.9 K 2023-04-23 05:56:26,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~383.81 KB/393024, currentSize=46.96 KB/48090 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 471ms, sequenceid=518, compaction requested=true 2023-04-23 05:56:26,473 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:26,473 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:26,473 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:56:26,473 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:56:26,473 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:26,474 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:26,475 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155168 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:56:26,476 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/A is initiating minor compaction (all files) 2023-04-23 05:56:26,476 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/A in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,476 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e132fc7418c341a19941f700eb2a1aa0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/666ed33d75b14d6f9cdc31e651597981, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/0f73c4baabac4c96a53909ccd13f612b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/074fbcaab1e440ccba815a93aac89059, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c3fee99ca6544ca59713813832ee919f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1210494ebb2949d0afca4cd5ab935c28] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=151.5 K 2023-04-23 05:56:26,476 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69572 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:56:26,476 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/B is initiating minor compaction (all files) 2023-04-23 05:56:26,476 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/B in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,476 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e48e414f6c274416817ca2d081198e55, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f4fd204e81954f1f88dd62d509898bd3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cd8c8e0871144720b98b118efb30325c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f841beaeb5f943a99ca59e612411dbcf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/46cc8b89c9d14031bf63ae031d41db44, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cc200a712c564041946e27a674c2e175] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=67.9 K 2023-04-23 05:56:26,477 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e132fc7418c341a19941f700eb2a1aa0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682229381924 2023-04-23 05:56:26,477 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e48e414f6c274416817ca2d081198e55, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682229381924 2023-04-23 05:56:26,477 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f4fd204e81954f1f88dd62d509898bd3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682229381998 2023-04-23 05:56:26,477 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 666ed33d75b14d6f9cdc31e651597981, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682229381998 2023-04-23 05:56:26,478 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting cd8c8e0871144720b98b118efb30325c, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682229382615 2023-04-23 05:56:26,478 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 0f73c4baabac4c96a53909ccd13f612b, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682229382615 2023-04-23 05:56:26,478 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f841beaeb5f943a99ca59e612411dbcf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682229383346 2023-04-23 05:56:26,478 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 46cc8b89c9d14031bf63ae031d41db44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1682229384410 2023-04-23 05:56:26,479 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting cc200a712c564041946e27a674c2e175, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1682229385979 2023-04-23 05:56:26,479 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 074fbcaab1e440ccba815a93aac89059, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682229383346 2023-04-23 05:56:26,480 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c3fee99ca6544ca59713813832ee919f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1682229384410 2023-04-23 05:56:26,480 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 1210494ebb2949d0afca4cd5ab935c28, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1682229385979 2023-04-23 05:56:26,495 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#B#compaction#525 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:26,500 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#A#compaction#526 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:26,524 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:26,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:26,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:26,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:26,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:26,528 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:26,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:26,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:26,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,528 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,530 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:26,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:26,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:26,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:26,531 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:26,531 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,531 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:26,532 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,532 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:26,532 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:56:26,532 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:26,532 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:26,532 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:26,532 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:26,532 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:26,532 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:26,563 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304237abe59cbbe29455cbc712d711bbcfc1f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304237abe59cbbe29455cbc712d711bbcfc1f 2023-04-23 05:56:26,586 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=528, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/20b0670e815d4f9b91f0928fce367b2c 2023-04-23 05:56:26,606 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/ea918a3838b34b89939f3bb3c2415f89 2023-04-23 05:56:26,626 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/18e4de8dc0d54a588df0dca188e54b6a 2023-04-23 05:56:26,630 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/20b0670e815d4f9b91f0928fce367b2c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/20b0670e815d4f9b91f0928fce367b2c 2023-04-23 05:56:26,634 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/20b0670e815d4f9b91f0928fce367b2c, entries=150, sequenceid=528, filesize=25.4 K 2023-04-23 05:56:26,635 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:26,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:26,635 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/ea918a3838b34b89939f3bb3c2415f89 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ea918a3838b34b89939f3bb3c2415f89 2023-04-23 05:56:26,641 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ea918a3838b34b89939f3bb3c2415f89, entries=150, sequenceid=528, filesize=11.9 K 2023-04-23 05:56:26,641 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/18e4de8dc0d54a588df0dca188e54b6a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/18e4de8dc0d54a588df0dca188e54b6a 2023-04-23 05:56:26,645 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/18e4de8dc0d54a588df0dca188e54b6a, entries=150, sequenceid=528, filesize=11.9 K 2023-04-23 05:56:26,645 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=6.71 KB/6870 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 113ms, sequenceid=528, compaction requested=true 2023-04-23 05:56:26,646 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:26,646 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:26,649 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:26,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3092926a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:26,652 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:26,652 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,653 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:26,653 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:26,653 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,653 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:26,655 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:26,655 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,655 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:26,655 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:26,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:26,658 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:26,658 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:26,658 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:26,658 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:26,658 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:26,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,659 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,660 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,662 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,664 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:26,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:26,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:26,664 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,665 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,665 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,665 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,665 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,665 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,665 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:26,665 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:26,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:26,669 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:56:26,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:26,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:26,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:26,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:26,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:26,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:26,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229386701, max timestamp=1682229386712Num uniques 859; 2023-04-23 05:56:26,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:26,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229386701, max timestamp=1682229386712Num uniques 859; 2023-04-23 05:56:26,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682229386701, max timestamp=1682229386712Num uniques 859; 2023-04-23 05:56:26,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:26,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:26,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229446759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,765 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:26,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229446761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:26,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:26,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:26,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:26,768 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:26,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:26,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:26,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,769 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,770 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,770 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:26,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:26,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:26,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:26,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,772 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:26,773 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,773 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:26,773 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:26,773 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:26,775 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,775 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,776 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,776 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:26,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2355fd3b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:26,777 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:26,777 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,778 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:26,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:26,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:26,779 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,779 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:26,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,780 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:26,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:26,782 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:26,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:26,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:26,783 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:26,783 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:26,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,784 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,784 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,786 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,791 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,791 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,791 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,791 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:26,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:26,791 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,791 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,791 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,791 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,791 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:26,791 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,792 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,791 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:26,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:26,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229446866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:26,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229446866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,892 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:26,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:26,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:26,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:26,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:26,895 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:26,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:26,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:26,896 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,896 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,897 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,897 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,897 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,897 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:26,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:26,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:26,898 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:26,899 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,899 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:26,899 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:26,899 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:26,899 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:26,900 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,900 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:26,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e679386[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:26,903 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:26,904 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,905 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,905 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:26,905 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:26,905 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:26,905 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,905 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:26,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,907 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:26,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:26,910 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:26,910 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:26,910 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:26,910 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:26,910 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:26,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,912 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:26,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:26,912 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,916 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:26,916 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:26,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:26,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:26,916 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,916 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:26,916 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:26,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:26,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:26,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:26,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:26,917 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:26,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:26,917 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:26,917 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:26,918 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/71f3f79d49324a7c8d2ca26359515087 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/71f3f79d49324a7c8d2ca26359515087 2023-04-23 05:56:26,922 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/B of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 71f3f79d49324a7c8d2ca26359515087(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:26,922 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:26,922 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, priority=10, startTime=1682229386473; duration=0sec 2023-04-23 05:56:26,922 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:26,922 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-23 05:56:26,924 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 81758 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-23 05:56:26,924 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/C is initiating minor compaction (all files) 2023-04-23 05:56:26,924 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/C in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:26,925 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9f2cc029d0074f3aa671d0cfd0a27695, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac5a59a919d14acf9e12fe33d1184e64, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/54101c3ecdf14aaa9fc86d7bf588bbb5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/81df9bcee09e4232aef36513ccf1cbe6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3a163f0ec3d44a4bbd6a808f24972624, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac9a7c7bad184c1f8fa25eaa1967e8cd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/18e4de8dc0d54a588df0dca188e54b6a] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=79.8 K 2023-04-23 05:56:26,925 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9f2cc029d0074f3aa671d0cfd0a27695, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1682229381924 2023-04-23 05:56:26,925 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ac5a59a919d14acf9e12fe33d1184e64, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1682229381998 2023-04-23 05:56:26,926 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 54101c3ecdf14aaa9fc86d7bf588bbb5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682229382615 2023-04-23 05:56:26,926 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 81df9bcee09e4232aef36513ccf1cbe6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1682229383346 2023-04-23 05:56:26,927 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3a163f0ec3d44a4bbd6a808f24972624, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1682229384410 2023-04-23 05:56:26,927 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ac9a7c7bad184c1f8fa25eaa1967e8cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1682229385979 2023-04-23 05:56:26,928 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 18e4de8dc0d54a588df0dca188e54b6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682229386010 2023-04-23 05:56:26,939 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#C#compaction#531 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:26,947 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/df0869251e8d43b691bad48b8eb57aa9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/df0869251e8d43b691bad48b8eb57aa9 2023-04-23 05:56:26,951 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/C of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into df0869251e8d43b691bad48b8eb57aa9(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:26,951 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:26,951 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, priority=9, startTime=1682229386474; duration=0sec 2023-04-23 05:56:26,951 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:27,017 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:27,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:27,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:27,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:27,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:27,021 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:27,021 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,021 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:27,021 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:27,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,022 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,023 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,024 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:27,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:27,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:27,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:27,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:27,025 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:27,028 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,028 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,029 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:27,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@305636ad[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:27,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:27,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,032 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:27,032 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:27,032 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:27,032 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,032 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:27,033 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,034 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:27,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:27,037 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:27,037 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:27,037 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:27,037 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:27,037 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:27,039 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,039 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,039 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,039 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,041 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,044 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,044 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,044 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,044 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:27,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:27,045 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,045 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,045 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,045 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,045 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,045 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,045 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:27,045 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:27,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:27,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229447070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229447075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,091 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d290a669a1814e22bce732dc74cfbf7b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d290a669a1814e22bce732dc74cfbf7b 2023-04-23 05:56:27,097 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=538, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/8ea3db4ce1b4469ba42cdcb768816843 2023-04-23 05:56:27,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/54dd6567b1dc45daa14b24ebd58c2e4f 2023-04-23 05:56:27,145 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:27,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:27,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:27,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:27,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:27,149 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:27,149 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,149 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:27,149 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:27,149 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,149 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,151 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,151 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,151 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,151 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:27,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:27,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:27,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:27,153 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,153 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:27,153 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:27,153 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,155 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14f3f30f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:27,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:27,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:27,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:27,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:27,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,159 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:27,162 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:27,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,162 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:27,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:27,165 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:27,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:27,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:27,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:27,165 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:27,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,167 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,168 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,170 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,172 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:27,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:27,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,172 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,172 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:27,173 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:27,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:27,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,273 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:27,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:27,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:27,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:27,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:27,277 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:27,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:27,277 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:27,278 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,278 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,279 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,279 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,279 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,279 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:27,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:27,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:27,285 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:27,285 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,285 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:27,285 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:27,285 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:27,286 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:27,291 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,291 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,292 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18c3788f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:27,293 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:27,294 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,295 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:27,295 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:27,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,295 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:27,296 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:27,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,296 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:27,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:27,299 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:27,299 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:27,299 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:27,299 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:27,299 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:27,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,300 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,301 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,302 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:27,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:27,305 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,305 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:27,306 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:27,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,306 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,306 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:27,335 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/918f74a83f644d0f80310d74a06f00a9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/918f74a83f644d0f80310d74a06f00a9 2023-04-23 05:56:27,340 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/A of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 918f74a83f644d0f80310d74a06f00a9(size=25.6 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:27,340 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:27,340 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, priority=10, startTime=1682229386473; duration=0sec 2023-04-23 05:56:27,340 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:27,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229447374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229447378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,406 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:27,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:27,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:27,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:27,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:27,410 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:27,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:27,410 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:27,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:27,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:27,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:27,414 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,414 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:27,414 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:27,414 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:27,414 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:27,416 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,416 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:27,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aaa23b2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:27,418 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:27,418 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,419 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:27,419 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:27,419 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,419 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:27,422 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:27,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,422 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:27,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:27,424 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:27,424 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:27,424 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:27,425 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:27,425 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:27,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,427 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,428 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,429 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,432 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:27,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:27,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,432 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,432 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:27,433 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:27,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229447488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,492 DEBUG [Thread-3841] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:27,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229447490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,494 DEBUG [Thread-3845] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:27,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229447495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,499 DEBUG [Thread-3843] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:27,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/3aabf6d85ed647c99b07ad9cbb4afb63 2023-04-23 05:56:27,533 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:27,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:27,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/8ea3db4ce1b4469ba42cdcb768816843 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/8ea3db4ce1b4469ba42cdcb768816843 2023-04-23 05:56:27,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:27,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:27,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:27,537 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:27,537 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,537 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:27,537 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:27,537 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,537 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,538 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/8ea3db4ce1b4469ba42cdcb768816843, entries=200, sequenceid=538, filesize=32.3 K 2023-04-23 05:56:27,539 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/54dd6567b1dc45daa14b24ebd58c2e4f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/54dd6567b1dc45daa14b24ebd58c2e4f 2023-04-23 05:56:27,539 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,539 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,539 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,539 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:27,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:27,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:27,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:27,541 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,541 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:27,541 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:27,541 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:27,541 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:27,542 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/54dd6567b1dc45daa14b24ebd58c2e4f, entries=150, sequenceid=538, filesize=11.9 K 2023-04-23 05:56:27,543 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/3aabf6d85ed647c99b07ad9cbb4afb63 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3aabf6d85ed647c99b07ad9cbb4afb63 2023-04-23 05:56:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,543 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,544 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:27,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bc88f14[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:27,546 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:27,546 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3aabf6d85ed647c99b07ad9cbb4afb63, entries=150, sequenceid=538, filesize=11.9 K 2023-04-23 05:56:27,547 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 878ms, sequenceid=538, compaction requested=true 2023-04-23 05:56:27,547 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:27,547 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:27,547 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:27,547 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:27,548 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:27,548 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:27,548 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:27,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:27,548 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:27,548 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,548 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:27,549 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85384 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:27,549 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:27,549 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/A is initiating minor compaction (all files) 2023-04-23 05:56:27,549 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/B is initiating minor compaction (all files) 2023-04-23 05:56:27,549 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/A in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,549 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/B in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,549 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/918f74a83f644d0f80310d74a06f00a9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/20b0670e815d4f9b91f0928fce367b2c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/8ea3db4ce1b4469ba42cdcb768816843] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=83.4 K 2023-04-23 05:56:27,549 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/71f3f79d49324a7c8d2ca26359515087, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ea918a3838b34b89939f3bb3c2415f89, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/54dd6567b1dc45daa14b24ebd58c2e4f] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=35.9 K 2023-04-23 05:56:27,549 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:27,549 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,549 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 918f74a83f644d0f80310d74a06f00a9, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1682229385979 2023-04-23 05:56:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:27,549 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 71f3f79d49324a7c8d2ca26359515087, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1682229385979 2023-04-23 05:56:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,550 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 20b0670e815d4f9b91f0928fce367b2c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682229386010 2023-04-23 05:56:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,550 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ea918a3838b34b89939f3bb3c2415f89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682229386010 2023-04-23 05:56:27,551 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 8ea3db4ce1b4469ba42cdcb768816843, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1682229386652 2023-04-23 05:56:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,551 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 54dd6567b1dc45daa14b24ebd58c2e4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1682229386652 2023-04-23 05:56:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:27,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:27,554 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:27,554 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:27,554 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:27,554 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:27,554 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:27,557 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,557 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:27,557 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:27,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:27,560 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,563 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#B#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:27,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:27,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:27,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:27,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:27,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:27,568 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:27,568 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:27,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:27,588 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#A#compaction#535 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:27,600 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/461d5f49a80d4af397555c1aa82aeb4f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/461d5f49a80d4af397555c1aa82aeb4f 2023-04-23 05:56:27,606 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/B of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 461d5f49a80d4af397555c1aa82aeb4f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:27,606 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:27,606 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, priority=13, startTime=1682229387547; duration=0sec 2023-04-23 05:56:27,606 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:27,606 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-23 05:56:27,607 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-23 05:56:27,607 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-23 05:56:27,607 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. because compaction request was cancelled 2023-04-23 05:56:27,668 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:27,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:27,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:27,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:27,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:27,672 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:27,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:27,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:27,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:27,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:27,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:27,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:27,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:27,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:27,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:27,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:27,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:27,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:27,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:27,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:27,676 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-23 05:56:27,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:27,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:27,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:27,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:27,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:27,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:27,693 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042376dba5731a904730ac9fdaa629e0aa58 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042376dba5731a904730ac9fdaa629e0aa58 2023-04-23 05:56:27,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:27,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:27,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229447894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229447896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:27,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:27,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229447998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:28,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229447999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,017 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/1c35c1e6ced142df840d15361759d768 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1c35c1e6ced142df840d15361759d768 2023-04-23 05:56:28,021 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/A of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 1c35c1e6ced142df840d15361759d768(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:28,021 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:28,021 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, priority=13, startTime=1682229387547; duration=0sec 2023-04-23 05:56:28,021 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:28,099 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=578, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/5b465749d5db4ba080d1cf6706e65b6d 2023-04-23 05:56:28,109 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/6c9f7e74c3c549528d25df6768ee8fb2 2023-04-23 05:56:28,118 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/b9b8ea7a06cf4863aee638401a532da1 2023-04-23 05:56:28,122 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/5b465749d5db4ba080d1cf6706e65b6d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/5b465749d5db4ba080d1cf6706e65b6d 2023-04-23 05:56:28,125 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/5b465749d5db4ba080d1cf6706e65b6d, entries=150, sequenceid=578, filesize=25.4 K 2023-04-23 05:56:28,126 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/6c9f7e74c3c549528d25df6768ee8fb2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/6c9f7e74c3c549528d25df6768ee8fb2 2023-04-23 05:56:28,129 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/6c9f7e74c3c549528d25df6768ee8fb2, entries=150, sequenceid=578, filesize=11.9 K 2023-04-23 05:56:28,131 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/b9b8ea7a06cf4863aee638401a532da1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b9b8ea7a06cf4863aee638401a532da1 2023-04-23 05:56:28,134 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b9b8ea7a06cf4863aee638401a532da1, entries=150, sequenceid=578, filesize=11.9 K 2023-04-23 05:56:28,135 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 459ms, sequenceid=578, compaction requested=true 2023-04-23 05:56:28,135 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:28,135 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,135 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:28,135 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:28,135 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,135 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:28,135 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:28,136 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,136 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,137 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:28,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a86ee9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:28,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:28,139 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,144 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,144 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:28,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:28,144 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:28,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:28,146 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:28,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,146 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:28,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:28,150 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:28,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:28,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:28,150 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:28,150 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:28,151 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,151 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,151 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,152 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,154 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:28,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:28,158 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,158 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:28,158 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:28,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:28,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:28,203 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:56:28,203 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:28,203 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:28,203 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:28,203 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:28,203 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:28,203 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:28,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:28,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229448233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229388228, max timestamp=1682229388235Num uniques 859; 2023-04-23 05:56:28,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229388228, max timestamp=1682229388235Num uniques 859; 2023-04-23 05:56:28,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:28,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:28,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229388228, max timestamp=1682229388235Num uniques 859; 2023-04-23 05:56:28,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:28,258 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:28,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:28,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:28,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:28,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:28,264 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:28,264 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,264 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:28,264 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:28,264 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,265 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,266 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,266 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,267 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,267 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:28,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:28,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:28,268 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:28,268 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,268 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:28,268 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:28,268 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:28,268 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,268 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:28,269 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:28,269 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,269 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:28,269 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:28,269 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,269 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:28,269 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:28,270 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,270 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,270 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:28,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ac6973[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:28,273 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:28,273 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,274 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,274 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:28,274 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:28,274 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,274 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:28,274 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:28,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:28,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,276 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:28,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:28,278 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:28,278 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:28,278 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:28,278 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:28,278 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:28,280 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,280 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,281 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,281 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,284 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,285 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,285 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,285 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,285 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,285 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,285 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,285 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,285 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:28,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:28,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:28,286 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:28,286 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:28,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:28,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229448287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:28,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229448337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,386 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:28,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:28,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:28,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:28,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:28,391 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:28,391 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,391 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:28,391 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:28,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,392 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,393 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,393 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,393 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,393 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:28,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229448390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:28,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:28,394 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:28,394 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,394 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:28,395 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:28,395 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:28,395 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,395 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:28,395 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:28,395 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,395 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:28,395 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:28,395 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,395 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:28,395 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:28,396 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,396 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,397 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,397 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:28,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58c06d07[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:28,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,400 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,400 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:28,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:28,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:28,401 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,401 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:28,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:28,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,402 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:28,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:28,404 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:28,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:28,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:28,404 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:28,404 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:28,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,407 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,408 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,411 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:28,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:28,411 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:28,411 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:28,511 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:28,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:28,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:28,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:28,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:28,515 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:28,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:28,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:28,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,517 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,517 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:28,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:28,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:28,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,518 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:28,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:28,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:28,519 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:28,519 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:28,520 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,520 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,521 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@173fe924[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:28,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:28,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,523 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,523 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:28,523 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:28,523 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:28,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:28,524 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:28,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,524 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:28,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:28,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:28,527 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:28,527 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:28,527 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:28,527 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:28,527 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:28,532 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,532 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,532 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,533 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,534 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,536 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:28,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:28,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,537 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:28,537 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:28,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:28,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229448541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:28,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229448595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,616 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d111be43ebfe428fb0710d0db4951ce4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d111be43ebfe428fb0710d0db4951ce4 2023-04-23 05:56:28,622 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=590, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/bfce5daf612f48b8bea76096f724978b 2023-04-23 05:56:28,637 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:28,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:28,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:28,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:28,645 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:28,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:28,645 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:28,646 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,646 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,647 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:28,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:28,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:28,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:28,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:28,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:28,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:28,649 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:28,650 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,650 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:28,650 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:28,650 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:28,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:28,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:28,650 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:28,651 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:28,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54a3020c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:28,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:28,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,656 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:28,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:28,656 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,657 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:28,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:28,658 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:28,658 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,658 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:28,658 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:28,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:28,661 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:28,661 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:28,661 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:28,662 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:28,662 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:28,663 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,663 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,663 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,664 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,665 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,668 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,668 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,668 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,668 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,668 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,668 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,669 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,669 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,669 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,669 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:28,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:28,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,669 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:28,669 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:28,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:28,769 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:28,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:28,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:28,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:28,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:28,773 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:28,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:28,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:28,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,776 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,776 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,776 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,776 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:28,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:28,778 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,778 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:28,778 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:28,778 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:28,778 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:28,779 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,780 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,780 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,780 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:28,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34240e1a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:28,781 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:28,781 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,782 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:28,782 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:28,782 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:28,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:28,783 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,783 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:28,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,784 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,784 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:28,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:28,786 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:28,786 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:28,786 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:28,786 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:28,786 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:28,789 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,789 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,789 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,791 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,793 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:28,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:28,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,794 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:28,794 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:28,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:28,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229448846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,894 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:28,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:28,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:28,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:28,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:28,898 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:28,898 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,898 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:28,898 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:28,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,900 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,900 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,900 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,900 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:28,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:28,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:28,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:28,901 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:28,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229448900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:28,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:28,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:28,902 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:28,902 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:28,904 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,904 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,904 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,904 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:28,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f57e5cc[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:28,906 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:28,906 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,909 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:28,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:28,909 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,909 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:28,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:28,910 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:28,910 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,910 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,910 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:28,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:28,913 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:28,913 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:28,913 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:28,913 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:28,913 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:28,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,915 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:28,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,916 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,917 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,922 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,922 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,922 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:28,922 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:28,922 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:28,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:28,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:28,923 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:28,923 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:28,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:28,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:28,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:28,923 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:28,923 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:28,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:28,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:29,023 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:29,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:29,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:29,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:29,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:29,027 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:29,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:29,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:29,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:29,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,027 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,028 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:29,028 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:29,029 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,029 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:29,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:29,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:29,031 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,031 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:29,031 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:29,031 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:29,032 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,032 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,033 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,033 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:29,033 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/07b3d5cff5794f3cbfe0d6ca35de6af3 2023-04-23 05:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:29,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74957cb[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:29,037 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:29,037 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,038 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:29,038 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,038 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:29,040 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,040 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:29,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:29,043 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:29,043 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:29,043 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:29,043 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:29,043 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:29,045 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,045 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:29,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,045 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,046 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/ee93f6ddb54343c8a06b71ada4a79931 2023-04-23 05:56:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,048 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:29,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:29,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:29,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:29,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:29,051 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:29,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,051 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:29,051 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:29,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:29,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/bfce5daf612f48b8bea76096f724978b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bfce5daf612f48b8bea76096f724978b 2023-04-23 05:56:29,051 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,058 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bfce5daf612f48b8bea76096f724978b, entries=150, sequenceid=590, filesize=25.4 K 2023-04-23 05:56:29,059 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/07b3d5cff5794f3cbfe0d6ca35de6af3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/07b3d5cff5794f3cbfe0d6ca35de6af3 2023-04-23 05:56:29,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/07b3d5cff5794f3cbfe0d6ca35de6af3, entries=100, sequenceid=590, filesize=9.5 K 2023-04-23 05:56:29,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/ee93f6ddb54343c8a06b71ada4a79931 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ee93f6ddb54343c8a06b71ada4a79931 2023-04-23 05:56:29,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ee93f6ddb54343c8a06b71ada4a79931, entries=100, sequenceid=590, filesize=9.5 K 2023-04-23 05:56:29,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 864ms, sequenceid=590, compaction requested=true 2023-04-23 05:56:29,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:29,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:29,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:29,067 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:29,067 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:29,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:29,068 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78442 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:29,068 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34422 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:29,068 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/A is initiating minor compaction (all files) 2023-04-23 05:56:29,068 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/B is initiating minor compaction (all files) 2023-04-23 05:56:29,068 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/A in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,068 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/B in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,068 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1c35c1e6ced142df840d15361759d768, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/5b465749d5db4ba080d1cf6706e65b6d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bfce5daf612f48b8bea76096f724978b] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=76.6 K 2023-04-23 05:56:29,069 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/461d5f49a80d4af397555c1aa82aeb4f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/6c9f7e74c3c549528d25df6768ee8fb2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/07b3d5cff5794f3cbfe0d6ca35de6af3] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=33.6 K 2023-04-23 05:56:29,069 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 1c35c1e6ced142df840d15361759d768, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1682229386652 2023-04-23 05:56:29,070 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 461d5f49a80d4af397555c1aa82aeb4f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1682229386652 2023-04-23 05:56:29,070 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 5b465749d5db4ba080d1cf6706e65b6d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1682229386733 2023-04-23 05:56:29,072 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 6c9f7e74c3c549528d25df6768ee8fb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1682229386733 2023-04-23 05:56:29,072 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting bfce5daf612f48b8bea76096f724978b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682229387886 2023-04-23 05:56:29,072 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 07b3d5cff5794f3cbfe0d6ca35de6af3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682229387894 2023-04-23 05:56:29,079 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#B#compaction#543 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:29,081 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#A#compaction#542 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:29,093 WARN [DataStreamer for file /user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/3decae56dcd54246b761fa6d254b5f81] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-23 05:56:29,098 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/3decae56dcd54246b761fa6d254b5f81 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/3decae56dcd54246b761fa6d254b5f81 2023-04-23 05:56:29,102 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/B of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 3decae56dcd54246b761fa6d254b5f81(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:29,102 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:29,102 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, priority=13, startTime=1682229389067; duration=0sec 2023-04-23 05:56:29,103 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:29,103 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:29,104 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46540 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:29,104 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/C is initiating minor compaction (all files) 2023-04-23 05:56:29,104 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/C in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,104 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/df0869251e8d43b691bad48b8eb57aa9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3aabf6d85ed647c99b07ad9cbb4afb63, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b9b8ea7a06cf4863aee638401a532da1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ee93f6ddb54343c8a06b71ada4a79931] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=45.4 K 2023-04-23 05:56:29,105 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting df0869251e8d43b691bad48b8eb57aa9, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1682229386010 2023-04-23 05:56:29,105 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3aabf6d85ed647c99b07ad9cbb4afb63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1682229386652 2023-04-23 05:56:29,105 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b9b8ea7a06cf4863aee638401a532da1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1682229386733 2023-04-23 05:56:29,105 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ee93f6ddb54343c8a06b71ada4a79931, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682229387894 2023-04-23 05:56:29,112 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#C#compaction#544 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:29,151 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:29,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:29,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:29,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:29,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:29,155 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:29,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:29,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:29,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:29,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,156 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:29,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:29,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,158 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,158 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:29,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:29,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:29,160 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-23 05:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:29,160 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:29,181 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423a7b1ff41622d49819c04b1be060f9ce5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a7b1ff41622d49819c04b1be060f9ce5 2023-04-23 05:56:29,186 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=627, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/d5511574cde141afb850986b3c4ff5bd 2023-04-23 05:56:29,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:29,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:29,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:29,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229449378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:29,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229449404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:29,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229449482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,523 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/ddf0c0a0e3984026aa2a3a91f8334ad1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ddf0c0a0e3984026aa2a3a91f8334ad1 2023-04-23 05:56:29,527 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/C of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into ddf0c0a0e3984026aa2a3a91f8334ad1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:29,527 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:29,527 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, priority=12, startTime=1682229389067; duration=0sec 2023-04-23 05:56:29,527 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:29,597 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/ee36e26e8ef4444b965de210528b6132 2023-04-23 05:56:29,607 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/626e79a2f62c49138809a4b76198868a 2023-04-23 05:56:29,612 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/d5511574cde141afb850986b3c4ff5bd as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/d5511574cde141afb850986b3c4ff5bd 2023-04-23 05:56:29,617 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/d5511574cde141afb850986b3c4ff5bd, entries=150, sequenceid=627, filesize=25.4 K 2023-04-23 05:56:29,617 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/ee36e26e8ef4444b965de210528b6132 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ee36e26e8ef4444b965de210528b6132 2023-04-23 05:56:29,621 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ee36e26e8ef4444b965de210528b6132, entries=150, sequenceid=627, filesize=11.9 K 2023-04-23 05:56:29,622 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/626e79a2f62c49138809a4b76198868a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/626e79a2f62c49138809a4b76198868a 2023-04-23 05:56:29,629 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/626e79a2f62c49138809a4b76198868a, entries=150, sequenceid=627, filesize=11.9 K 2023-04-23 05:56:29,630 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 470ms, sequenceid=627, compaction requested=false 2023-04-23 05:56:29,630 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:29,630 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:29,630 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:29,632 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,632 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b36864c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:29,635 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:29,635 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,636 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,636 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:29,636 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:29,636 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:29,636 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,637 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:29,638 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:29,644 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:29,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:29,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:29,644 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:29,644 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:29,645 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,645 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:29,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,645 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:29,647 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:29,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:29,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:29,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:29,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:29,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,652 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,652 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:29,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:29,652 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:29,653 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:29,688 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:56:29,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:29,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:29,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:29,688 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:29,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:29,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:29,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:29,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229449750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,753 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:29,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:29,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:29,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:29,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:29,756 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:29,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:29,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:29,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:29,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,758 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:29,758 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:29,758 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,758 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:29,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:29,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:29,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:29,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:29,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:29,760 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,760 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:29,760 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:29,760 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,761 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ff615ee[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:29,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:29,764 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,765 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,765 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,765 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,765 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:29,765 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:29,765 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:29,768 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:29,768 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:29,771 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:29,771 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:29,771 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:29,771 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:29,771 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:29,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,773 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:29,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:29,773 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,775 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,779 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,779 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,779 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:29,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:29,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:29,780 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:29,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:29,780 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:29,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,780 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:29,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:29,780 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:29,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:29,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:29,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229449853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,880 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:29,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:29,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:29,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:29,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:29,883 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:29,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:29,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:29,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:29,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,884 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,886 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,886 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:29,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,887 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,887 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:29,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:29,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:29,888 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,888 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:29,888 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:29,888 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:29,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:29,890 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,890 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,890 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,890 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:29,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:29,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f5fe259[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:29,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:29,892 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,893 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:29,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,893 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:29,893 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:29,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:29,899 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:29,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:29,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:29,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:29,899 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:29,905 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,905 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,905 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:29,906 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:29,908 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:29,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:29,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:29,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:29,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:29,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:29,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:29,912 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:29,912 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:29,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:29,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:29,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:29,912 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:29,912 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:29,932 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/bc52048609bf4a13a501a4bc1ab7e367 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bc52048609bf4a13a501a4bc1ab7e367 2023-04-23 05:56:29,936 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/A of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into bc52048609bf4a13a501a4bc1ab7e367(size=25.8 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:29,936 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:29,936 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, priority=13, startTime=1682229389067; duration=0sec 2023-04-23 05:56:29,936 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:30,012 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:30,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:30,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:30,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:30,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:30,015 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:30,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:30,015 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:30,016 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,016 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,017 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,017 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:30,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:30,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:30,019 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,019 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:30,019 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:30,019 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:30,019 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,020 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,021 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,021 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:30,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42e50b76[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:30,023 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,025 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:30,026 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,026 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:30,027 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:30,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:30,030 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:30,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:30,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:30,030 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:30,030 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:30,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,031 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,032 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,036 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:30,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:30,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,037 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:30,037 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:30,036 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,037 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,037 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,037 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:30,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:30,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229450057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,112 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423e41b8997405c45e5b49a40ee314e53e6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e41b8997405c45e5b49a40ee314e53e6 2023-04-23 05:56:30,134 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=640, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/261d3bf6299347b49d55479a470fc4af 2023-04-23 05:56:30,137 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:30,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:30,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:30,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:30,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:30,142 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:30,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:30,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:30,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,142 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,144 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,144 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,145 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,145 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:30,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:30,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:30,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:30,147 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,147 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:30,147 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:30,147 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:30,149 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,149 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,149 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,149 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:30,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:30,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d9ef180[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:30,151 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:30,151 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,152 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,152 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:30,152 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:30,152 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,152 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:30,152 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:30,154 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:30,157 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:30,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:30,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:30,157 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:30,157 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:30,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,159 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,161 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:30,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:30,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,163 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,164 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:30,164 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:30,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:30,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,164 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,264 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:30,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:30,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:30,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:30,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:30,268 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:30,268 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,268 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:30,268 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:30,268 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,269 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,270 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,270 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,270 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,270 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:30,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:30,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:30,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:30,271 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:30,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:30,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:30,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:30,272 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,272 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:30,272 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:30,272 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:30,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:30,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:30,272 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:30,278 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,278 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,279 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,279 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5228dab8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:30,280 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:30,280 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,281 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,281 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:30,282 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:30,282 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:30,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:30,283 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,283 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:30,283 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,283 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:30,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:30,285 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:30,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:30,285 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:30,286 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:30,286 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:30,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,288 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,290 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,293 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:30,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:30,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,294 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:30,294 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:30,294 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,294 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,294 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,294 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:30,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229450358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,394 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:30,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:30,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:30,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:30,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:30,397 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:30,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:30,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:30,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,399 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,399 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,400 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,400 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:30,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:30,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:30,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:30,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:30,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:30,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:30,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:30,401 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,402 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:30,402 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:30,402 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:30,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:30,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:30,402 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:30,403 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:30,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2215b78e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:30,406 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:30,406 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,407 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:30,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:30,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:30,407 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,407 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:30,410 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,410 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,410 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,410 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:30,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:30,412 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:30,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:30,412 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:30,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:30,413 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:30,414 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,414 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,414 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,415 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229450414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,416 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,419 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,419 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:30,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:30,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,419 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,419 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:30,419 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:30,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,519 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:30,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:30,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:30,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:30,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:30,523 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:30,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:30,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:30,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,524 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,525 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,525 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:30,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:30,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:30,526 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,526 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:30,526 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:30,526 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:30,528 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,529 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:30,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4841b601[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:30,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:30,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,531 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,531 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:30,531 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:30,531 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:30,531 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:30,533 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:30,533 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,533 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,533 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:30,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:30,535 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:30,535 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:30,536 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:30,536 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:30,536 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:30,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:30,537 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:30,538 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,539 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:30,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:30,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:30,542 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:30,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:30,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,543 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:30,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,543 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:30,543 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:30,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:30,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:30,547 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/f0e75a55959b4d10bfa0bb444fe4783e 2023-04-23 05:56:30,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/9599e9d7b7454c0c900b6f496a7ef167 2023-04-23 05:56:30,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/261d3bf6299347b49d55479a470fc4af as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/261d3bf6299347b49d55479a470fc4af 2023-04-23 05:56:30,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/261d3bf6299347b49d55479a470fc4af, entries=150, sequenceid=640, filesize=25.4 K 2023-04-23 05:56:30,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/f0e75a55959b4d10bfa0bb444fe4783e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f0e75a55959b4d10bfa0bb444fe4783e 2023-04-23 05:56:30,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f0e75a55959b4d10bfa0bb444fe4783e, entries=150, sequenceid=640, filesize=11.9 K 2023-04-23 05:56:30,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/9599e9d7b7454c0c900b6f496a7ef167 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9599e9d7b7454c0c900b6f496a7ef167 2023-04-23 05:56:30,573 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9599e9d7b7454c0c900b6f496a7ef167, entries=150, sequenceid=640, filesize=11.9 K 2023-04-23 05:56:30,574 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 886ms, sequenceid=640, compaction requested=true 2023-04-23 05:56:30,574 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:30,574 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:30,574 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:30,574 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-23 05:56:30,574 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:30,574 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:30,575 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:30,575 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/B is initiating minor compaction (all files) 2023-04-23 05:56:30,575 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/B in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,576 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/3decae56dcd54246b761fa6d254b5f81, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ee36e26e8ef4444b965de210528b6132, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f0e75a55959b4d10bfa0bb444fe4783e] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=36.1 K 2023-04-23 05:56:30,576 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78548 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:30,576 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/A is initiating minor compaction (all files) 2023-04-23 05:56:30,576 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/A in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,576 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bc52048609bf4a13a501a4bc1ab7e367, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/d5511574cde141afb850986b3c4ff5bd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/261d3bf6299347b49d55479a470fc4af] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=76.7 K 2023-04-23 05:56:30,578 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 3decae56dcd54246b761fa6d254b5f81, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682229386756 2023-04-23 05:56:30,578 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting bc52048609bf4a13a501a4bc1ab7e367, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682229386756 2023-04-23 05:56:30,578 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ee36e26e8ef4444b965de210528b6132, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682229388273 2023-04-23 05:56:30,578 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting d5511574cde141afb850986b3c4ff5bd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682229388273 2023-04-23 05:56:30,578 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting f0e75a55959b4d10bfa0bb444fe4783e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229389369 2023-04-23 05:56:30,579 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 261d3bf6299347b49d55479a470fc4af, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229389369 2023-04-23 05:56:30,586 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#B#compaction#551 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:30,592 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#A#compaction#552 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:30,633 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/d66ad551af1f44558e8640231d60d6f9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d66ad551af1f44558e8640231d60d6f9 2023-04-23 05:56:30,640 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/B of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into d66ad551af1f44558e8640231d60d6f9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:30,640 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:30,640 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, priority=13, startTime=1682229390574; duration=0sec 2023-04-23 05:56:30,640 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:30,640 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:30,641 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:30,641 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/C is initiating minor compaction (all files) 2023-04-23 05:56:30,641 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/C in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,641 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ddf0c0a0e3984026aa2a3a91f8334ad1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/626e79a2f62c49138809a4b76198868a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9599e9d7b7454c0c900b6f496a7ef167] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=36.1 K 2023-04-23 05:56:30,642 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting ddf0c0a0e3984026aa2a3a91f8334ad1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1682229386756 2023-04-23 05:56:30,642 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 626e79a2f62c49138809a4b76198868a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1682229388273 2023-04-23 05:56:30,643 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 9599e9d7b7454c0c900b6f496a7ef167, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229389369 2023-04-23 05:56:30,643 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:30,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:30,644 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/cde241e8a5e94625bd1d95c65b7b373b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cde241e8a5e94625bd1d95c65b7b373b 2023-04-23 05:56:30,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:30,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:30,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:30,648 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:30,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:30,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:30,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:30,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:30,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:30,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:30,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:30,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:30,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:30,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:30,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:30,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:30,651 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:30,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:30,652 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:30,652 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:30,652 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#C#compaction#553 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:30,652 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:30,652 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:56:30,652 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/A of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into cde241e8a5e94625bd1d95c65b7b373b(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:30,652 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:30,653 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, priority=13, startTime=1682229390574; duration=0sec 2023-04-23 05:56:30,653 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:30,653 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:30,653 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:30,653 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:30,653 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:30,653 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:30,654 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:30,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:30,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:30,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:30,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229450888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:30,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:30,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229450991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,063 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/339db8f12259461782b41f392bd0f5c8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/339db8f12259461782b41f392bd0f5c8 2023-04-23 05:56:31,067 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/C of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 339db8f12259461782b41f392bd0f5c8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:31,067 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:31,067 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, priority=13, startTime=1682229390574; duration=0sec 2023-04-23 05:56:31,067 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:31,070 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042379e8350ae3aa4af2953668c197b8d244 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042379e8350ae3aa4af2953668c197b8d244 2023-04-23 05:56:31,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229451194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,475 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=663, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/af43f4ab11624195aed3f3c295967593 2023-04-23 05:56:31,484 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/511d447291334dfba505f7b1a132e81b 2023-04-23 05:56:31,493 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/fe46b98c9fab4f07ab88560bec95e929 2023-04-23 05:56:31,497 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/af43f4ab11624195aed3f3c295967593 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/af43f4ab11624195aed3f3c295967593 2023-04-23 05:56:31,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229451496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,500 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/af43f4ab11624195aed3f3c295967593, entries=150, sequenceid=663, filesize=25.4 K 2023-04-23 05:56:31,501 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/511d447291334dfba505f7b1a132e81b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/511d447291334dfba505f7b1a132e81b 2023-04-23 05:56:31,504 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/511d447291334dfba505f7b1a132e81b, entries=150, sequenceid=663, filesize=11.9 K 2023-04-23 05:56:31,505 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/fe46b98c9fab4f07ab88560bec95e929 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fe46b98c9fab4f07ab88560bec95e929 2023-04-23 05:56:31,508 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fe46b98c9fab4f07ab88560bec95e929, entries=150, sequenceid=663, filesize=11.9 K 2023-04-23 05:56:31,509 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 857ms, sequenceid=663, compaction requested=false 2023-04-23 05:56:31,509 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:31,509 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:31,509 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:31,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:31,511 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-23 05:56:31,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:31,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:31,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:31,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:31,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:31,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2194c78[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:31,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:31,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,526 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:31,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:31,527 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,527 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:31,528 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,528 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,528 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:31,528 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:31,532 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:31,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:31,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:31,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:31,532 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:31,532 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d372843221db4a64af581b3b27f1fa77 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d372843221db4a64af581b3b27f1fa77 2023-04-23 05:56:31,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,534 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,534 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,537 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,538 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=677, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/ecb8d6497b2c455ba907c90e67723a7b 2023-04-23 05:56:31,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229451537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,541 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:31,541 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:31,541 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,541 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:31,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:31,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:31,541 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:31,541 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:31,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:31,541 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:31,542 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:31,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,542 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229451540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229451541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/4709f33fcdb64c79a1884337edb6ade1 2023-04-23 05:56:31,559 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=677 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/a1ab04f8eef3498b84225144d5307bf1 2023-04-23 05:56:31,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/ecb8d6497b2c455ba907c90e67723a7b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ecb8d6497b2c455ba907c90e67723a7b 2023-04-23 05:56:31,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ecb8d6497b2c455ba907c90e67723a7b, entries=150, sequenceid=677, filesize=25.4 K 2023-04-23 05:56:31,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/4709f33fcdb64c79a1884337edb6ade1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/4709f33fcdb64c79a1884337edb6ade1 2023-04-23 05:56:31,570 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/4709f33fcdb64c79a1884337edb6ade1, entries=150, sequenceid=677, filesize=11.9 K 2023-04-23 05:56:31,571 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/a1ab04f8eef3498b84225144d5307bf1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/a1ab04f8eef3498b84225144d5307bf1 2023-04-23 05:56:31,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/a1ab04f8eef3498b84225144d5307bf1, entries=150, sequenceid=677, filesize=11.9 K 2023-04-23 05:56:31,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 64ms, sequenceid=677, compaction requested=true 2023-04-23 05:56:31,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:31,575 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:31,575 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:31,575 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:31,575 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:31,575 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:31,576 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78650 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:31,576 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/A is initiating minor compaction (all files) 2023-04-23 05:56:31,577 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:31,577 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/A in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,577 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/B is initiating minor compaction (all files) 2023-04-23 05:56:31,577 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cde241e8a5e94625bd1d95c65b7b373b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/af43f4ab11624195aed3f3c295967593, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ecb8d6497b2c455ba907c90e67723a7b] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=76.8 K 2023-04-23 05:56:31,577 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/B in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,577 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d66ad551af1f44558e8640231d60d6f9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/511d447291334dfba505f7b1a132e81b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/4709f33fcdb64c79a1884337edb6ade1] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=36.2 K 2023-04-23 05:56:31,577 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting cde241e8a5e94625bd1d95c65b7b373b, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229389369 2023-04-23 05:56:31,577 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d66ad551af1f44558e8640231d60d6f9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229389369 2023-04-23 05:56:31,578 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting af43f4ab11624195aed3f3c295967593, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682229389733 2023-04-23 05:56:31,578 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 511d447291334dfba505f7b1a132e81b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682229389733 2023-04-23 05:56:31,578 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting ecb8d6497b2c455ba907c90e67723a7b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1682229390880 2023-04-23 05:56:31,578 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 4709f33fcdb64c79a1884337edb6ade1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1682229390880 2023-04-23 05:56:31,584 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#B#compaction#560 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:31,589 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#A#compaction#561 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:31,609 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/013a6676c88c40f9b6f177f11cd2b916 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/013a6676c88c40f9b6f177f11cd2b916 2023-04-23 05:56:31,616 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/B of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 013a6676c88c40f9b6f177f11cd2b916(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:31,616 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:31,616 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, priority=13, startTime=1682229391575; duration=0sec 2023-04-23 05:56:31,616 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:31,616 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-23 05:56:31,618 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-23 05:56:31,618 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/C is initiating minor compaction (all files) 2023-04-23 05:56:31,618 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/C in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,618 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/339db8f12259461782b41f392bd0f5c8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fe46b98c9fab4f07ab88560bec95e929, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/a1ab04f8eef3498b84225144d5307bf1] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=36.2 K 2023-04-23 05:56:31,619 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 339db8f12259461782b41f392bd0f5c8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1682229389369 2023-04-23 05:56:31,619 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting fe46b98c9fab4f07ab88560bec95e929, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1682229389733 2023-04-23 05:56:31,619 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting a1ab04f8eef3498b84225144d5307bf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1682229390880 2023-04-23 05:56:31,631 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#C#compaction#562 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:31,642 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:31,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:31,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:31,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:56:31,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:31,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:31,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:31,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:31,645 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:31,645 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:31,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:31,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:31,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:31,648 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:31,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:31,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:31,648 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:31,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,650 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:31,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:31,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:31,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229451657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229451657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229451659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:31,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:31,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:31,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:31,668 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:31,674 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/e59fa9c210dc47e8be486af76699ee70 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e59fa9c210dc47e8be486af76699ee70 2023-04-23 05:56:31,678 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/C of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into e59fa9c210dc47e8be486af76699ee70(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:31,678 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:31,678 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, priority=13, startTime=1682229391575; duration=0sec 2023-04-23 05:56:31,678 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:31,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:31,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:31,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:31,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:31,679 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:31,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:31,679 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:31,680 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:31,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,682 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:31,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16f0bb99[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:31,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:31,684 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,685 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:31,686 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:31,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,686 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:31,689 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:31,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,689 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:31,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:31,692 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:31,692 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:31,692 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:31,692 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:31,692 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:31,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,693 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,694 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,696 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:31,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:31,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,698 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:31,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:31,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:31,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:31,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:31,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,698 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,698 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:31,699 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:31,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:31,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:31,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229451760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229451761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229451763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,799 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:31,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:31,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:31,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:31,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:31,802 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:31,802 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:31,802 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:31,802 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:31,802 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,802 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,804 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:31,804 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:31,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:31,804 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,804 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:31,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:31,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:31,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:31,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:31,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:31,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:31,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:31,805 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:31,806 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,806 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:31,806 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:31,806 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,806 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:31,806 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:31,806 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,806 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:31,806 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:31,808 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,809 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:31,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:31,810 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:31,810 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19d6c70e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:31,811 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:31,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:31,811 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:31,811 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,811 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:31,813 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,813 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:31,813 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:31,813 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:31,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:31,816 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:31,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:31,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:31,816 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:31,816 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:31,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,817 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:31,818 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,819 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,822 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:31,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:31,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:31,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:31,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:31,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:31,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:31,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:31,823 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:31,823 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:31,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,823 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:31,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:31,923 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:31,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:31,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:31,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:31,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:31,928 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:31,928 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:31,928 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:31,928 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:31,928 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,928 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,930 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:31,930 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,930 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,930 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:31,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:31,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:31,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:31,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:31,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:31,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:31,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:31,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:31,935 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:31,935 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,935 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:31,935 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:31,936 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:31,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:31,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:31,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:31,936 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:31,937 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,937 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:31,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ce28b8b[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:31,939 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:31,939 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,940 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:31,940 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,940 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:31,950 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,950 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,950 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:31,950 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:31,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:31,953 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:31,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:31,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:31,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:31,953 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:31,955 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:31,955 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,956 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:31,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:31,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:31,956 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,958 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,961 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:31,961 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:31,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:31,961 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:31,961 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,961 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:31,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:31,961 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:31,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:31,962 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:31,962 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:31,962 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:31,962 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,962 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:31,962 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:31,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229451964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229451964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:31,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:31,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229451966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:32,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229452002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,043 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/c77bb5709f3f44b59af173a7fba9c09a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c77bb5709f3f44b59af173a7fba9c09a 2023-04-23 05:56:32,048 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/A of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into c77bb5709f3f44b59af173a7fba9c09a(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:32,048 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:32,049 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, priority=13, startTime=1682229391575; duration=0sec 2023-04-23 05:56:32,049 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:32,062 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:32,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:32,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:32,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:32,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:32,065 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:32,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:32,065 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:32,066 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,066 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,073 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,073 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,073 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,073 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:32,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:32,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,074 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:32,075 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,075 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:32,075 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:32,075 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:32,075 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,077 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:32,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:32,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14d066e3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:32,079 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:32,080 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,081 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,081 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:32,081 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:32,081 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:32,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,081 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:32,082 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:32,082 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,082 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,082 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:32,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:32,086 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:32,086 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:32,086 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:32,086 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:32,086 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:32,087 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d3e5ff1f4c7a4cc6be81eec272875c89 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d3e5ff1f4c7a4cc6be81eec272875c89 2023-04-23 05:56:32,089 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,089 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,089 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,090 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,092 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,093 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=698, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/47f49d9193b54a32b34c3250202226c4 2023-04-23 05:56:32,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,095 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,095 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:32,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:32,095 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,095 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,096 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,096 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,096 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,096 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,096 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,096 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,097 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:32,097 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:32,197 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:32,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:32,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:32,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:32,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:32,205 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:32,205 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,205 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:32,205 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:32,205 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,205 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,206 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,207 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,207 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,207 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:32,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:32,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:32,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:32,208 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:32,208 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,208 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:32,208 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:32,208 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:32,208 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,208 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:32,208 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:32,208 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:32,210 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,210 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,210 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,210 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:32,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:32,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef2b93[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:32,212 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:32,212 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,216 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,216 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:32,216 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:32,216 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,216 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:32,216 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:32,218 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,218 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:32,218 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,218 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:32,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:32,221 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:32,221 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:32,221 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:32,221 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:32,221 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:32,223 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,223 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,223 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,224 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,226 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,228 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:32,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:32,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,228 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:32,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,228 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:32,228 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:32,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:32,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229452267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:32,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229452268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:32,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229452269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,329 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:32,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:32,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:32,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:32,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:32,333 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:32,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:32,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:32,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,333 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,335 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,336 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,336 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:32,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:32,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:32,337 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,337 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:32,338 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:32,338 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,338 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:32,338 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:32,338 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,338 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:32,338 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:32,339 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,339 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,339 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,340 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:32,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ae62d7[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:32,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:32,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,342 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:32,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:32,342 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:32,342 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,342 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:32,343 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,343 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:32,344 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,344 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:32,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:32,346 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:32,346 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:32,346 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:32,346 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:32,346 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:32,348 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,348 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,348 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,348 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,350 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,354 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:32,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:32,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:32,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,356 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,356 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,356 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,356 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,356 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:32,356 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:32,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:32,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229452425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,428 DEBUG [Thread-3837] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., hostname=jenkins-hbase4.apache.org,40995,1682229223026, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-23 05:56:32,456 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:32,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:32,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:32,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:32,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:32,460 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:32,460 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,460 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:32,460 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:32,461 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,461 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,464 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,464 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,464 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,464 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:32,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:32,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:32,466 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,466 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:32,466 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:32,466 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:32,466 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:32,468 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,468 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,468 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,468 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:32,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:32,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d80a9a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:32,470 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,471 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,472 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:32,472 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:32,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,472 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:32,472 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,474 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,475 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:32,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:32,477 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:32,477 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:32,477 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:32,477 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:32,477 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:32,479 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,479 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,480 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,480 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,481 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,484 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:32,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:32,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:32,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,484 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:32,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,484 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:32,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,484 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/b2c0ffc65cf44f5db971c087d82f54ae 2023-04-23 05:56:32,584 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:32,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:32,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:32,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:32,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:32,588 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:32,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:32,588 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:32,589 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,589 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,590 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,590 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,591 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,591 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:32,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:32,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:32,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:32,592 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,592 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:32,592 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:32,592 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:32,594 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,595 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,595 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,596 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:32,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77288f1c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:32,597 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:32,597 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,598 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,598 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:32,599 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:32,599 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:32,598 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,599 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:32,600 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,600 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:32,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:32,604 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:32,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:32,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:32,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:32,604 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:32,606 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,606 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,606 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,607 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,609 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,612 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,612 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,612 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,612 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,612 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,612 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,612 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:32,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:32,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,612 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,613 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,613 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,613 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:32,613 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:32,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,713 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:32,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:32,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:32,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:32,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:32,716 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:32,716 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,716 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:32,716 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:32,717 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,717 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,718 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,718 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,718 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,719 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:32,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:32,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:32,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:32,719 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:32,719 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:32,720 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,720 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:32,720 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:32,720 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:32,720 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:32,721 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,721 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,722 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,722 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:32,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96591d9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:32,723 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:32,723 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,724 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:32,724 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,725 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:32,724 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:32,725 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:32,727 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:32,727 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,727 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,727 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:32,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:32,729 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:32,729 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:32,729 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:32,729 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:32,729 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:32,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,731 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,731 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,733 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,735 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,735 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,735 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,735 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,735 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,735 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:32,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:32,735 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,736 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:32,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:32,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,736 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:32,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229452769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:32,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229452773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:32,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229452777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,836 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:32,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:32,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:32,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:32,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:32,840 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:32,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:32,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:32,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,840 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,843 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,843 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,843 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,843 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:32,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:32,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:32,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,844 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:32,845 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,845 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:32,845 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:32,845 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:32,846 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,846 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,847 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,847 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:32,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5286f2f8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:32,849 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:32,849 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,850 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,850 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:32,850 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:32,850 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:32,851 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,851 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:32,852 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,852 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:32,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:32,855 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:32,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:32,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:32,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:32,855 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:32,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,858 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:32,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,858 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,860 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:32,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,862 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:32,862 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:32,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:32,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:32,863 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:32,863 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:32,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/c6e27f1777e44776b21c74363bc56128 2023-04-23 05:56:32,919 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/47f49d9193b54a32b34c3250202226c4 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/47f49d9193b54a32b34c3250202226c4 2023-04-23 05:56:32,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/47f49d9193b54a32b34c3250202226c4, entries=200, sequenceid=698, filesize=32.3 K 2023-04-23 05:56:32,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/b2c0ffc65cf44f5db971c087d82f54ae as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/b2c0ffc65cf44f5db971c087d82f54ae 2023-04-23 05:56:32,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/b2c0ffc65cf44f5db971c087d82f54ae, entries=150, sequenceid=698, filesize=11.9 K 2023-04-23 05:56:32,927 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/c6e27f1777e44776b21c74363bc56128 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/c6e27f1777e44776b21c74363bc56128 2023-04-23 05:56:32,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/c6e27f1777e44776b21c74363bc56128, entries=150, sequenceid=698, filesize=11.9 K 2023-04-23 05:56:32,931 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 1287ms, sequenceid=698, compaction requested=false 2023-04-23 05:56:32,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:32,963 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:32,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:32,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:32,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:32,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:32,967 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:32,967 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,967 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:32,967 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:32,967 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,967 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,969 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:32,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:32,969 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,969 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:32,969 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:32,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:32,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:32,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:32,970 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:32,971 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:32,971 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:32,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:32,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:32,971 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:56:32,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:32,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:32,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:32,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:32,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:32,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:33,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:33,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:33,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:33,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229453063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:33,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:33,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229453168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:33,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:33,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229453370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:33,389 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304238cac9f1a7b0c468a8b6520eef528f958 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304238cac9f1a7b0c468a8b6520eef528f958 2023-04-23 05:56:33,394 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=712, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/668085a0c84f487dae6c6d2cfdebf330 2023-04-23 05:56:33,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:33,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229453675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:33,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:33,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229453772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:33,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:33,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229453783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:33,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:33,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229453785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:33,803 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/aed1ef2c2bb04b918da8d2557317da31 2023-04-23 05:56:34,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:34,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229454180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,212 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/1db33ab66f924ab3babfdacb45a02d9a 2023-04-23 05:56:34,215 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/668085a0c84f487dae6c6d2cfdebf330 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/668085a0c84f487dae6c6d2cfdebf330 2023-04-23 05:56:34,219 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/668085a0c84f487dae6c6d2cfdebf330, entries=150, sequenceid=712, filesize=25.4 K 2023-04-23 05:56:34,220 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/aed1ef2c2bb04b918da8d2557317da31 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aed1ef2c2bb04b918da8d2557317da31 2023-04-23 05:56:34,223 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aed1ef2c2bb04b918da8d2557317da31, entries=150, sequenceid=712, filesize=11.9 K 2023-04-23 05:56:34,223 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/1db33ab66f924ab3babfdacb45a02d9a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1db33ab66f924ab3babfdacb45a02d9a 2023-04-23 05:56:34,226 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1db33ab66f924ab3babfdacb45a02d9a, entries=150, sequenceid=712, filesize=11.9 K 2023-04-23 05:56:34,227 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 1256ms, sequenceid=712, compaction requested=true 2023-04-23 05:56:34,227 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:34,227 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:34,227 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:34,229 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,229 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:34,229 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:34,229 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:34,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:34,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722aaee8[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:34,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:34,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:34,236 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:34,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:34,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:34,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:34,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:34,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,237 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:34,237 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:34,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:34,237 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:34,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:34,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:34,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:34,240 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:34,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:34,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:34,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:34,240 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:34,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:34,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:34,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:34,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:34,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:34,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:34,243 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:34,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:34,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:34,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,246 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:34,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:34,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:34,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:34,246 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,246 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:34,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:34,247 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:34,246 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:34,247 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:34,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:34,246 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:34,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:34,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:34,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:34,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:34,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:34,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:34,347 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:34,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:34,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:34,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:34,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:34,352 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:34,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:34,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:34,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:34,353 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:34,353 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:34,355 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:34,355 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:34,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:34,356 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:34,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:34,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:34,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:34,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:34,357 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:34,357 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:34,357 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-23 05:56:34,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:34,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:34,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:34,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:34,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:34,358 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:34,773 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423821164724bb84866b98c04cc3ac9ce8a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423821164724bb84866b98c04cc3ac9ce8a 2023-04-23 05:56:34,778 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=732, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e320934b8fa54e3eaa3f19f4f66fc606 2023-04-23 05:56:35,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:35,188 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:35,188 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/15f43fca6f8b445998527fa81d433856 2023-04-23 05:56:35,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229455211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 462 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229455314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229455518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,599 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=732 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/781050eeccfe4353b751f6f0effc33a9 2023-04-23 05:56:35,602 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e320934b8fa54e3eaa3f19f4f66fc606 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e320934b8fa54e3eaa3f19f4f66fc606 2023-04-23 05:56:35,605 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e320934b8fa54e3eaa3f19f4f66fc606, entries=150, sequenceid=732, filesize=25.4 K 2023-04-23 05:56:35,606 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/15f43fca6f8b445998527fa81d433856 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/15f43fca6f8b445998527fa81d433856 2023-04-23 05:56:35,609 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/15f43fca6f8b445998527fa81d433856, entries=150, sequenceid=732, filesize=11.9 K 2023-04-23 05:56:35,610 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/781050eeccfe4353b751f6f0effc33a9 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/781050eeccfe4353b751f6f0effc33a9 2023-04-23 05:56:35,613 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/781050eeccfe4353b751f6f0effc33a9, entries=150, sequenceid=732, filesize=11.9 K 2023-04-23 05:56:35,614 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 1257ms, sequenceid=732, compaction requested=true 2023-04-23 05:56:35,614 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:35,614 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:35,615 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:35,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:35,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:35,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:35,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:35,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c99fde[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:35,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:35,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:35,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:35,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:35,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:35,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:35,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:35,619 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:35,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,619 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:35,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:35,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:35,620 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,620 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:35,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:35,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:35,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:35,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:35,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:35,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:35,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:35,623 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:35,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:35,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:35,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:35,623 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:35,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:35,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:35,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:35,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:35,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:35,626 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:35,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:35,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:35,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,628 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:35,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:35,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:35,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:35,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:35,630 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:35,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:35,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:35,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:35,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:35,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:35,630 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:35,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:35,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:35,630 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:35,630 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:35,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:35,730 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:35,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:35,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:35,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:35,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:35,734 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:35,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:35,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:35,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:35,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:35,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:35,740 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,740 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:35,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:35,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:35,740 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,740 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:35,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:35,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:35,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:35,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:35,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:35,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:35,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:35,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:35,742 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:35,742 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:35,742 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:35,742 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:35,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:35,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:35,742 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:56:35,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:35,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:35,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:35,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:35,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:35,742 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:35,760 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042348771a579dac4d438ed2ed632bb5ae47 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042348771a579dac4d438ed2ed632bb5ae47 2023-04-23 05:56:35,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:35,793 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:35,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229455816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229455819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229455820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 466 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229455822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229455921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229455923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:35,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:35,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229455924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229456124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229456126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229456127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,173 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=744, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e7af765886eb4c3285587aec8fe07b00 2023-04-23 05:56:36,184 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/d27f7db0e95f4f2f8de15455d699ff6e 2023-04-23 05:56:36,193 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=744 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/7cceb233c8e24842811620e8898ced4f 2023-04-23 05:56:36,197 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e7af765886eb4c3285587aec8fe07b00 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e7af765886eb4c3285587aec8fe07b00 2023-04-23 05:56:36,200 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e7af765886eb4c3285587aec8fe07b00, entries=150, sequenceid=744, filesize=25.4 K 2023-04-23 05:56:36,201 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/d27f7db0e95f4f2f8de15455d699ff6e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d27f7db0e95f4f2f8de15455d699ff6e 2023-04-23 05:56:36,204 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d27f7db0e95f4f2f8de15455d699ff6e, entries=150, sequenceid=744, filesize=11.9 K 2023-04-23 05:56:36,205 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/7cceb233c8e24842811620e8898ced4f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/7cceb233c8e24842811620e8898ced4f 2023-04-23 05:56:36,208 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/7cceb233c8e24842811620e8898ced4f, entries=150, sequenceid=744, filesize=11.9 K 2023-04-23 05:56:36,209 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 467ms, sequenceid=744, compaction requested=true 2023-04-23 05:56:36,209 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:36,209 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:36,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:36,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:36,209 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:36,211 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,211 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,211 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,211 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:36,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64073567[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:36,213 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:36,213 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,214 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:36,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:36,214 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:36,214 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,214 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:36,215 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,215 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:36,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:36,217 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:36,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:36,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:36,218 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:36,218 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:36,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,230 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,231 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,232 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,234 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,234 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,234 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,234 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:36,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:36,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:36,234 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,234 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,235 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,235 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:36,235 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,235 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:36,235 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,235 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:36,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-23 05:56:36,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:36,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:36,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:36,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:36,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:36,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:36,335 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:36,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:36,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:36,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:36,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:36,341 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:36,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:36,341 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:36,342 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,342 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,343 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,343 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,344 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,344 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:36,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:36,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:36,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:36,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:36,345 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:36,345 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d9a6c92cb0d041c286a4e430e3ef9f96 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d9a6c92cb0d041c286a4e430e3ef9f96 2023-04-23 05:56:36,346 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,346 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:36,346 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:36,346 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:36,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:36,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:36,346 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:36,347 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,347 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,348 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:36,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1730a2d[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:36,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:36,349 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,350 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=765, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/848e50e7b159413683f0a8e3553d8c6d 2023-04-23 05:56:36,351 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,352 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:36,352 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:36,352 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:36,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,352 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:36,353 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:36,353 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,353 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:36,353 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:36,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:36,358 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:36,359 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:36,359 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:36,359 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:36,359 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:36,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=765 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/25b8b27df5f347dc94ff2f5abf83c4a3 2023-04-23 05:56:36,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,361 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,362 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,363 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229456361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=765 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/1e9db7dbb6604e79a0eebcdd14045b55 2023-04-23 05:56:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:36,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:36,371 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,372 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,372 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,372 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,372 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,372 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:36,372 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:36,371 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:36,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/848e50e7b159413683f0a8e3553d8c6d as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/848e50e7b159413683f0a8e3553d8c6d 2023-04-23 05:56:36,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/848e50e7b159413683f0a8e3553d8c6d, entries=200, sequenceid=765, filesize=32.3 K 2023-04-23 05:56:36,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/25b8b27df5f347dc94ff2f5abf83c4a3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/25b8b27df5f347dc94ff2f5abf83c4a3 2023-04-23 05:56:36,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/25b8b27df5f347dc94ff2f5abf83c4a3, entries=150, sequenceid=765, filesize=11.9 K 2023-04-23 05:56:36,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/1e9db7dbb6604e79a0eebcdd14045b55 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1e9db7dbb6604e79a0eebcdd14045b55 2023-04-23 05:56:36,386 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1e9db7dbb6604e79a0eebcdd14045b55, entries=150, sequenceid=765, filesize=11.9 K 2023-04-23 05:56:36,387 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 60ms, sequenceid=765, compaction requested=true 2023-04-23 05:56:36,387 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:36,387 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:36,387 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:36,387 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:56:36,387 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:36,387 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:56:36,389 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 170970 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:56:36,389 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/A is initiating minor compaction (all files) 2023-04-23 05:56:36,389 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/A in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,389 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c77bb5709f3f44b59af173a7fba9c09a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/47f49d9193b54a32b34c3250202226c4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/668085a0c84f487dae6c6d2cfdebf330, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e320934b8fa54e3eaa3f19f4f66fc606, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e7af765886eb4c3285587aec8fe07b00, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/848e50e7b159413683f0a8e3553d8c6d] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=167.0 K 2023-04-23 05:56:36,390 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:56:36,390 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/B is initiating minor compaction (all files) 2023-04-23 05:56:36,390 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/B in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,390 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/013a6676c88c40f9b6f177f11cd2b916, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/b2c0ffc65cf44f5db971c087d82f54ae, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aed1ef2c2bb04b918da8d2557317da31, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/15f43fca6f8b445998527fa81d433856, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d27f7db0e95f4f2f8de15455d699ff6e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/25b8b27df5f347dc94ff2f5abf83c4a3] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=72.0 K 2023-04-23 05:56:36,390 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c77bb5709f3f44b59af173a7fba9c09a, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1682229390880 2023-04-23 05:56:36,390 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 013a6676c88c40f9b6f177f11cd2b916, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1682229390880 2023-04-23 05:56:36,391 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 47f49d9193b54a32b34c3250202226c4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1682229391535 2023-04-23 05:56:36,391 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b2c0ffc65cf44f5db971c087d82f54ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1682229391535 2023-04-23 05:56:36,391 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 668085a0c84f487dae6c6d2cfdebf330, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1682229391652 2023-04-23 05:56:36,391 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting aed1ef2c2bb04b918da8d2557317da31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1682229391652 2023-04-23 05:56:36,391 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e320934b8fa54e3eaa3f19f4f66fc606, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1682229393039 2023-04-23 05:56:36,392 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 15f43fca6f8b445998527fa81d433856, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1682229393039 2023-04-23 05:56:36,392 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e7af765886eb4c3285587aec8fe07b00, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1682229395194 2023-04-23 05:56:36,392 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d27f7db0e95f4f2f8de15455d699ff6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1682229395194 2023-04-23 05:56:36,393 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 848e50e7b159413683f0a8e3553d8c6d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=765, earliestPutTs=1682229395814 2023-04-23 05:56:36,393 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 25b8b27df5f347dc94ff2f5abf83c4a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=765, earliestPutTs=1682229395814 2023-04-23 05:56:36,404 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#B#compaction#579 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:36,414 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#A#compaction#578 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:36,422 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/616dbdf6af14451c9555be5d8efc03b8 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/616dbdf6af14451c9555be5d8efc03b8 2023-04-23 05:56:36,427 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/B of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 616dbdf6af14451c9555be5d8efc03b8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:36,427 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:36,427 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, priority=10, startTime=1682229396387; duration=0sec 2023-04-23 05:56:36,427 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:36,427 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-23 05:56:36,428 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73696 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-23 05:56:36,428 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/C is initiating minor compaction (all files) 2023-04-23 05:56:36,428 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/C in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,429 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e59fa9c210dc47e8be486af76699ee70, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/c6e27f1777e44776b21c74363bc56128, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1db33ab66f924ab3babfdacb45a02d9a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/781050eeccfe4353b751f6f0effc33a9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/7cceb233c8e24842811620e8898ced4f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1e9db7dbb6604e79a0eebcdd14045b55] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=72.0 K 2023-04-23 05:56:36,429 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e59fa9c210dc47e8be486af76699ee70, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=677, earliestPutTs=1682229390880 2023-04-23 05:56:36,429 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c6e27f1777e44776b21c74363bc56128, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1682229391535 2023-04-23 05:56:36,430 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1db33ab66f924ab3babfdacb45a02d9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1682229391652 2023-04-23 05:56:36,430 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 781050eeccfe4353b751f6f0effc33a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=732, earliestPutTs=1682229393039 2023-04-23 05:56:36,430 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 7cceb233c8e24842811620e8898ced4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=744, earliestPutTs=1682229395194 2023-04-23 05:56:36,431 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1e9db7dbb6604e79a0eebcdd14045b55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=765, earliestPutTs=1682229395814 2023-04-23 05:56:36,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:36,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-23 05:56:36,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:36,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:36,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:36,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:36,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:36,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:36,453 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#C#compaction#581 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682229396454, max timestamp=1682229396463Num uniques 859; 2023-04-23 05:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682229396454, max timestamp=1682229396463Num uniques 859; 2023-04-23 05:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682229396454, max timestamp=1682229396463Num uniques 859; 2023-04-23 05:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:36,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-23 05:56:36,472 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:36,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:36,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:36,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:36,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:36,476 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:36,476 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,476 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:36,476 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:36,477 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,477 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,479 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,479 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,479 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,479 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:36,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:36,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229456476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,481 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:36,481 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,481 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:36,481 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:36,481 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:36,482 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,482 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:36,482 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:36,482 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,482 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:36,482 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:36,482 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,482 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:36,482 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:36,483 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,483 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,484 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229456480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229456481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:36,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,485 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:36,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2609a905[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:36,486 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229456481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229456482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,487 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:36,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:36,487 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:36,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,487 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:36,488 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:36,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,488 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:36,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:36,491 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:36,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:36,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:36,491 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:36,491 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:36,493 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,493 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,493 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,494 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,495 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,497 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:36,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:36,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:36,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,497 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,497 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:36,498 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:36,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229456582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229456586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229456586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229456586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229456587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,598 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:36,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:36,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:36,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:36,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:36,602 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:36,602 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,602 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:36,602 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:36,603 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,603 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,604 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,604 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,605 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:36,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:36,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:36,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:36,606 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,606 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:36,606 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:36,606 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:36,606 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:36,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,608 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,608 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:36,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb3e575[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:36,609 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:36,610 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,614 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:36,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:36,614 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:36,614 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,614 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:36,615 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:36,615 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,616 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:36,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:36,618 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:36,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:36,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:36,618 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:36,618 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:36,619 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,620 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,620 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,621 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,622 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,626 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:36,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:36,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,626 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,626 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:36,626 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,626 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:36,626 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,626 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,726 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:36,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:36,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:36,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:36,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:36,730 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:36,730 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,730 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:36,730 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:36,730 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,730 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,732 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:36,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:36,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:36,733 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:36,733 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:36,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:36,734 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,734 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:36,734 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:36,734 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:36,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,734 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:36,734 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:36,736 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,736 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:36,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:36,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c24169[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:36,738 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:36,738 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,739 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:36,739 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:36,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,739 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:36,741 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,741 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:36,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:36,744 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:36,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:36,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:36,744 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:36,744 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:36,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,745 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,746 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,750 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:36,749 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:36,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:36,750 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,750 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,750 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,750 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:36,750 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:36,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229456785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229456790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229456790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229456790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:36,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229456791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,850 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:36,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:36,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:36,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:36,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:36,853 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:36,853 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,854 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:36,854 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:36,854 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,854 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,855 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,855 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:36,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:36,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:36,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:36,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,856 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:36,857 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:36,857 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:36,857 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,857 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:36,857 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:36,857 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,857 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:36,857 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:36,857 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,857 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:36,857 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:36,859 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,859 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,860 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,860 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423e1364125c91e4b17b86391bd67608912 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e1364125c91e4b17b86391bd67608912 2023-04-23 05:56:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:36,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff91ffe[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:36,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:36,863 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,864 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:36,864 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,864 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:36,865 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:36,865 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,865 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,865 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:36,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,866 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/3fed367826d7470e9fe3683cd44f22d1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3fed367826d7470e9fe3683cd44f22d1 2023-04-23 05:56:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,868 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=778, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/1994d6c3926749e8866c5e78ff1c51ba 2023-04-23 05:56:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:36,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:36,868 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:36,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:36,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:36,868 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:36,868 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:36,870 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,871 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,871 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,873 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,876 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/C of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 3fed367826d7470e9fe3683cd44f22d1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:36,876 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:36,876 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, priority=10, startTime=1682229396387; duration=0sec 2023-04-23 05:56:36,876 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:36,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,877 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,878 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:36,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:36,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:36,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,878 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,878 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:36,878 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:36,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,896 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=778 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/68c43aae920a43efaf3f1f8e42de1885 2023-04-23 05:56:36,978 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:36,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:36,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:36,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:36,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:36,982 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:36,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:36,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:36,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,982 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,983 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:36,983 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:36,984 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,984 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:36,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:36,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:36,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:36,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:36,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:36,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:36,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:36,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:36,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:36,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:36,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:36,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453c285[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:36,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:36,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:36,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:36,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:36,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:36,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:36,995 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,995 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,995 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:36,995 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:36,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:36,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:36,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:36,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:36,997 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:36,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:36,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:36,997 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:36,997 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:36,999 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,999 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,999 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:36,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:36,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,001 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:37,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:37,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:37,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:37,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:37,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:37,004 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:37,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,004 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:37,004 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:37,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:37,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:37,004 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229457089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229457094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229457094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,096 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229457095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229457095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,104 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:37,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:37,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:37,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:37,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:37,108 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:37,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:37,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:37,108 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:37,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,109 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,111 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:37,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,111 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:37,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:37,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:37,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:37,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:37,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:37,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:37,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:37,112 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:37,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:37,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:37,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:37,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:37,113 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:37,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:37,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:37,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:37,113 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:37,115 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:37,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,115 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:37,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:37,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:37,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@307f1737[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:37,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:37,117 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,118 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,118 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:37,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:37,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:37,118 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:37,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:37,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:37,119 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:37,119 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:37,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:37,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:37,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:37,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:37,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:37,122 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:37,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:37,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:37,122 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:37,122 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:37,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:37,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:37,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,125 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:37,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:37,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:37,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:37,126 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,127 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,129 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:37,129 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:37,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:37,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:37,129 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,129 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:37,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:37,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:37,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,130 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,130 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:37,130 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:37,130 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:37,230 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:37,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:37,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:37,234 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/35dba8b3043940718fb81dc758f9e1f0 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/35dba8b3043940718fb81dc758f9e1f0 2023-04-23 05:56:37,239 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/A of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 35dba8b3043940718fb81dc758f9e1f0(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:37,239 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:37,239 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, priority=10, startTime=1682229396387; duration=0sec 2023-04-23 05:56:37,239 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:37,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:37,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:37,242 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:37,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:37,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:37,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:37,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,242 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,243 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:37,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:37,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:37,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:37,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:37,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:37,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:37,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:37,245 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:37,245 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:37,245 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:37,245 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:37,245 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,246 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:37,246 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:37,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:37,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,247 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:37,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:37,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:37,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5452c158[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:37,249 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:37,249 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,251 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,251 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:37,251 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:37,251 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:37,251 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,251 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:37,253 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:37,253 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:37,253 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,253 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:37,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:37,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:37,256 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:37,256 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:37,256 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:37,256 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:37,256 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:37,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:37,257 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:37,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:37,258 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:37,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,259 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:37,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:37,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,262 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:37,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:37,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:37,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:37,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:37,262 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,263 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:37,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:37,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:37,263 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:37,263 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:37,263 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:37,263 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:37,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:37,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=778 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/0eb178baeda24943b728250862ef1ac1 2023-04-23 05:56:37,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/1994d6c3926749e8866c5e78ff1c51ba as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1994d6c3926749e8866c5e78ff1c51ba 2023-04-23 05:56:37,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1994d6c3926749e8866c5e78ff1c51ba, entries=200, sequenceid=778, filesize=32.3 K 2023-04-23 05:56:37,343 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/68c43aae920a43efaf3f1f8e42de1885 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/68c43aae920a43efaf3f1f8e42de1885 2023-04-23 05:56:37,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/68c43aae920a43efaf3f1f8e42de1885, entries=150, sequenceid=778, filesize=11.9 K 2023-04-23 05:56:37,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/0eb178baeda24943b728250862ef1ac1 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/0eb178baeda24943b728250862ef1ac1 2023-04-23 05:56:37,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/0eb178baeda24943b728250862ef1ac1, entries=150, sequenceid=778, filesize=11.9 K 2023-04-23 05:56:37,351 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 919ms, sequenceid=778, compaction requested=false 2023-04-23 05:56:37,351 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:37,363 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:37,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:37,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:37,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:37,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:37,366 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:37,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:37,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:37,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:37,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,366 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,370 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:37,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:37,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,370 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:37,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:37,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:37,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:37,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:37,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:37,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:37,371 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:37,372 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:37,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:37,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:37,372 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-23 05:56:37,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:37,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:37,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:37,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:37,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:37,372 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:37,389 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423c5b6ef03dd2e4ba3afb1f32a8bd60431 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c5b6ef03dd2e4ba3afb1f32a8bd60431 2023-04-23 05:56:37,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:37,599 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:37,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229457603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229457604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229457604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229457605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 492 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229457606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229457708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229457708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229457710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229457710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 494 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229457710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,798 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=815, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/ad3aa4d6ce8646aea81ccc102f787767 2023-04-23 05:56:37,808 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/995330346b5148c494e38e665943a45b 2023-04-23 05:56:37,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229457911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229457912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229457913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,915 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229457913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:37,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:37,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 496 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229457914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55906 deadline: 1682229458215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55894 deadline: 1682229458215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,218 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/76c8a3d80b944482a7dc5748e33acc0a 2023-04-23 05:56:38,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 498 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229458219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,222 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/ad3aa4d6ce8646aea81ccc102f787767 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ad3aa4d6ce8646aea81ccc102f787767 2023-04-23 05:56:38,226 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ad3aa4d6ce8646aea81ccc102f787767, entries=150, sequenceid=815, filesize=25.4 K 2023-04-23 05:56:38,227 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/995330346b5148c494e38e665943a45b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/995330346b5148c494e38e665943a45b 2023-04-23 05:56:38,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229458226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,231 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/995330346b5148c494e38e665943a45b, entries=150, sequenceid=815, filesize=11.9 K 2023-04-23 05:56:38,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229458228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,232 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/76c8a3d80b944482a7dc5748e33acc0a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/76c8a3d80b944482a7dc5748e33acc0a 2023-04-23 05:56:38,235 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/76c8a3d80b944482a7dc5748e33acc0a, entries=150, sequenceid=815, filesize=11.9 K 2023-04-23 05:56:38,236 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 864ms, sequenceid=815, compaction requested=true 2023-04-23 05:56:38,236 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:38,236 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,236 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:38,237 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:38,237 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,237 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:38,237 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:38,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,238 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,238 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:38,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@737322c3[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:38,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:38,240 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,241 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:38,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:38,241 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:38,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,241 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:38,242 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,242 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:38,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:38,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:38,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:38,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:38,244 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:38,244 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:38,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,247 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,248 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,249 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:38,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:38,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,254 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:38,254 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,254 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:38,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,254 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:38,354 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:38,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:38,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:38,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:38,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:38,358 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:38,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:38,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:38,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,358 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,360 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,360 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,360 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,360 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:38,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:38,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:38,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:38,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:38,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,361 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:38,362 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:38,362 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:38,362 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,362 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:38,362 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-23 05:56:38,362 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:38,362 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:38,362 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:38,362 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:38,362 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:38,362 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:38,374 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423b0a336854fcd49c9b47a8919c5e3dedf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423b0a336854fcd49c9b47a8919c5e3dedf 2023-04-23 05:56:38,381 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=826, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e3ef5bf66f504177b51ab882db1be852 2023-04-23 05:56:38,392 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=826 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/18a2e509efad4307b8f96afa9eff7025 2023-04-23 05:56:38,403 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=826 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/334ac44ecdd14aaca051b639b6cd12c3 2023-04-23 05:56:38,407 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e3ef5bf66f504177b51ab882db1be852 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e3ef5bf66f504177b51ab882db1be852 2023-04-23 05:56:38,411 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e3ef5bf66f504177b51ab882db1be852, entries=150, sequenceid=826, filesize=25.4 K 2023-04-23 05:56:38,412 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/18a2e509efad4307b8f96afa9eff7025 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/18a2e509efad4307b8f96afa9eff7025 2023-04-23 05:56:38,415 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/18a2e509efad4307b8f96afa9eff7025, entries=150, sequenceid=826, filesize=11.9 K 2023-04-23 05:56:38,416 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/334ac44ecdd14aaca051b639b6cd12c3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/334ac44ecdd14aaca051b639b6cd12c3 2023-04-23 05:56:38,420 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/334ac44ecdd14aaca051b639b6cd12c3, entries=150, sequenceid=826, filesize=11.9 K 2023-04-23 05:56:38,421 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 59ms, sequenceid=826, compaction requested=true 2023-04-23 05:56:38,421 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:38,421 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:38,421 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:38,423 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,423 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,423 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,423 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:38,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:38,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69ae81c2[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:38,425 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:38,425 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,429 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,429 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:38,430 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,430 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:38,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:38,431 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,431 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:38,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:38,434 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:38,434 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:38,434 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:38,434 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:38,434 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:38,435 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,435 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,435 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,437 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,438 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:38,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:38,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,440 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,441 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:38,440 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,441 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:38,441 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,441 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,541 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:38,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:38,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:38,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:38,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:38,545 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:38,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:38,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:38,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,547 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,547 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,548 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,548 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:38,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,549 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:38,550 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,550 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:38,550 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:38,550 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:38,550 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:38,551 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,552 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:38,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2120f56f[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:38,553 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:38,554 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,554 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:38,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:38,555 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:38,555 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,555 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:38,556 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,556 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:38,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:38,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:38,558 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:38,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:38,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:38,558 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:38,558 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:38,564 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,564 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,564 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,564 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,566 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,568 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:38,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:38,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,568 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:38,568 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:38,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,568 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,668 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:38,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:38,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:38,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:38,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:38,672 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:38,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:38,672 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:38,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,674 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:38,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:38,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:38,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:38,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:38,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:38,675 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:38,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:38,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:38,676 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:38,676 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:38,677 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,677 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,678 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:38,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63bab812[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:38,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:38,679 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,681 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:38,681 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:38,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,681 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:38,683 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,683 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:38,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:38,685 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:38,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:38,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:38,685 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:38,685 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:38,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,687 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,688 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,689 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:38,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:38,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:38,695 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:38,695 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:38,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,695 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,695 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,696 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:38,751 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:56:38,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:38,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:38,751 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:38,751 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:38,752 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:38,752 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:38,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229398767, max timestamp=1682229398774Num uniques 859; 2023-04-23 05:56:38,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:56:38,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682229398767, max timestamp=1682229398774Num uniques 859; 2023-04-23 05:56:38,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-23 05:56:38,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229398767, max timestamp=1682229398774Num uniques 859; 2023-04-23 05:56:38,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:38,781 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423bc90b0d021a44d7089a29b8964d290d0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423bc90b0d021a44d7089a29b8964d290d0 2023-04-23 05:56:38,788 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=839, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/9eb6c8d0cbee418d8de8329a9dae926b 2023-04-23 05:56:38,795 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:38,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:38,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229458791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 508 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229458792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:38,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229458793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229458797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229458798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:38,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:38,804 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:38,804 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,804 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:38,804 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:38,804 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,806 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,806 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:38,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:38,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:38,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,807 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:38,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:38,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:38,808 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,808 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:38,808 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:38,808 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:38,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:38,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:38,808 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:38,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,810 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:38,810 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:38,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@935eb9c[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:38,812 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:38,812 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,814 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,814 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:38,814 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:38,814 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:38,815 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:38,816 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,816 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:38,816 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,816 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:38,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:38,819 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:38,819 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:38,819 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:38,819 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:38,819 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:38,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,820 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,821 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,823 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,825 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,826 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:38,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:38,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,826 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:38,826 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:38,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,826 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:38,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229458899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229458899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 510 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229458899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:55868 deadline: 1682229458904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:38,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229458904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,926 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:38,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:38,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:38,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:38,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:38,930 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:38,930 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,930 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:38,930 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:38,930 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,931 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,936 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,936 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,937 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,937 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:38,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:38,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:38,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:38,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:38,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:38,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:38,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:38,938 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:38,938 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,938 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:38,939 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:38,939 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:38,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:38,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:38,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:38,939 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:38,940 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,940 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,941 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,941 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:38,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1408cbe9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:38,943 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:38,943 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,944 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:38,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:38,944 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:38,944 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,944 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:38,945 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:38,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,945 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:38,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:38,948 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:38,949 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:38,949 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:38,949 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:38,949 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:38,952 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,952 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,952 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:38,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:38,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:38,953 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,954 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:38,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:38,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,956 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:38,956 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:38,957 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,957 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,957 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:38,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:38,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:38,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:38,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:38,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:38,957 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:38,957 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:38,957 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,957 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:38,957 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:38,957 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:38,957 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:38,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:38,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:39,057 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:39,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:39,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:39,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:39,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:39,061 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:39,061 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:39,061 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:39,061 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:39,062 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,062 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,063 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,063 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:39,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:39,063 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,063 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:39,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:39,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:39,065 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,065 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:39,065 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:39,065 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:39,065 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:39,066 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,066 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:39,066 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:39,067 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,067 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,068 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,068 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:39,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3225cf8a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:39,069 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:39,069 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,071 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,071 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:39,071 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:39,071 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:39,071 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,071 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:39,072 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,072 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,072 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:39,073 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:39,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:39,075 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:39,075 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:39,075 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:39,075 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:39,075 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:39,077 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,077 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,077 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:39,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:39,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,078 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,079 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:39,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:39,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:39,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:39,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:39,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:39,083 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:39,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,083 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:39,083 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:39,083 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:39,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:39,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229459104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229459106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 512 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229459106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55868 deadline: 1682229459108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229459111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,183 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:39,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:39,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:39,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:39,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:39,187 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:39,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:39,187 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:39,188 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:39,188 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,188 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,193 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:39,193 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:39,193 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,193 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:39,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:39,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:39,194 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:39,194 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,194 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:39,195 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:39,195 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,195 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:39,195 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:39,195 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,195 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:39,195 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:39,196 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,196 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:39,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:39,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e98bf4[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:39,198 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:39,198 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,199 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,199 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:39,199 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:39,199 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:39,199 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,199 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:39,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/bbefc3594b1b400994475dd35a59c6f5 2023-04-23 05:56:39,202 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,202 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:39,202 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:39,202 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:39,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:39,205 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:39,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:39,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:39,206 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:39,206 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:39,208 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,208 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:39,208 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:39,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:39,209 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,210 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:39,212 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:39,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/9617f6c2bc73462994f33a83901ee070 2023-04-23 05:56:39,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:39,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:39,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:39,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:39,212 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,213 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,213 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,213 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,213 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,213 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,213 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:39,213 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:39,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:39,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:39,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:39,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/9eb6c8d0cbee418d8de8329a9dae926b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/9eb6c8d0cbee418d8de8329a9dae926b 2023-04-23 05:56:39,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/9eb6c8d0cbee418d8de8329a9dae926b, entries=300, sequenceid=839, filesize=46.1 K 2023-04-23 05:56:39,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/bbefc3594b1b400994475dd35a59c6f5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/bbefc3594b1b400994475dd35a59c6f5 2023-04-23 05:56:39,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/bbefc3594b1b400994475dd35a59c6f5, entries=150, sequenceid=839, filesize=11.9 K 2023-04-23 05:56:39,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/9617f6c2bc73462994f33a83901ee070 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9617f6c2bc73462994f33a83901ee070 2023-04-23 05:56:39,226 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9617f6c2bc73462994f33a83901ee070, entries=150, sequenceid=839, filesize=11.9 K 2023-04-23 05:56:39,227 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=136.42 KB/139690 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 476ms, sequenceid=839, compaction requested=true 2023-04-23 05:56:39,227 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:39,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:39,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:39,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:39,227 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:56:39,227 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:56:39,228 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 159212 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:56:39,229 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:56:39,229 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/A is initiating minor compaction (all files) 2023-04-23 05:56:39,229 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/B is initiating minor compaction (all files) 2023-04-23 05:56:39,229 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/A in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,229 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/B in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,229 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/35dba8b3043940718fb81dc758f9e1f0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1994d6c3926749e8866c5e78ff1c51ba, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ad3aa4d6ce8646aea81ccc102f787767, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e3ef5bf66f504177b51ab882db1be852, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/9eb6c8d0cbee418d8de8329a9dae926b] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=155.5 K 2023-04-23 05:56:39,229 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/616dbdf6af14451c9555be5d8efc03b8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/68c43aae920a43efaf3f1f8e42de1885, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/995330346b5148c494e38e665943a45b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/18a2e509efad4307b8f96afa9eff7025, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/bbefc3594b1b400994475dd35a59c6f5] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=60.3 K 2023-04-23 05:56:39,229 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 35dba8b3043940718fb81dc758f9e1f0, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=765, earliestPutTs=1682229395814 2023-04-23 05:56:39,229 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 616dbdf6af14451c9555be5d8efc03b8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=765, earliestPutTs=1682229395814 2023-04-23 05:56:39,230 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 68c43aae920a43efaf3f1f8e42de1885, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=778, earliestPutTs=1682229396340 2023-04-23 05:56:39,230 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 1994d6c3926749e8866c5e78ff1c51ba, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=778, earliestPutTs=1682229396340 2023-04-23 05:56:39,230 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 995330346b5148c494e38e665943a45b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1682229396473 2023-04-23 05:56:39,230 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting ad3aa4d6ce8646aea81ccc102f787767, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1682229396473 2023-04-23 05:56:39,230 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 18a2e509efad4307b8f96afa9eff7025, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=826, earliestPutTs=1682229397602 2023-04-23 05:56:39,231 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e3ef5bf66f504177b51ab882db1be852, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=826, earliestPutTs=1682229397602 2023-04-23 05:56:39,231 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting bbefc3594b1b400994475dd35a59c6f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1682229398748 2023-04-23 05:56:39,231 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 9eb6c8d0cbee418d8de8329a9dae926b, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1682229398741 2023-04-23 05:56:39,241 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#B#compaction#593 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:39,249 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#A#compaction#594 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:39,282 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/cb828972c2624bf4b8ae47a0bf76388e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cb828972c2624bf4b8ae47a0bf76388e 2023-04-23 05:56:39,287 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/A of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into cb828972c2624bf4b8ae47a0bf76388e(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:39,287 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:39,287 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, priority=11, startTime=1682229399227; duration=0sec 2023-04-23 05:56:39,287 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:39,287 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-23 05:56:39,289 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-23 05:56:39,289 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/C is initiating minor compaction (all files) 2023-04-23 05:56:39,289 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/C in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,289 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3fed367826d7470e9fe3683cd44f22d1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/0eb178baeda24943b728250862ef1ac1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/76c8a3d80b944482a7dc5748e33acc0a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/334ac44ecdd14aaca051b639b6cd12c3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9617f6c2bc73462994f33a83901ee070] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=60.3 K 2023-04-23 05:56:39,289 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 3fed367826d7470e9fe3683cd44f22d1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=765, earliestPutTs=1682229395814 2023-04-23 05:56:39,290 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 0eb178baeda24943b728250862ef1ac1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=778, earliestPutTs=1682229396340 2023-04-23 05:56:39,290 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 76c8a3d80b944482a7dc5748e33acc0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1682229396473 2023-04-23 05:56:39,291 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 334ac44ecdd14aaca051b639b6cd12c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=826, earliestPutTs=1682229397602 2023-04-23 05:56:39,291 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 9617f6c2bc73462994f33a83901ee070, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1682229398748 2023-04-23 05:56:39,299 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#C#compaction#595 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:39,314 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:39,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:39,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:39,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:39,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:39,317 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:39,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:39,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:39,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:39,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,317 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,318 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:39,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:39,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:39,318 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,319 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,319 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:39,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:39,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:39,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:39,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:39,320 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:39,320 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,320 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:39,320 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-04-23 05:56:39,321 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:39,321 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:39,321 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:39,321 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:39,321 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:39,321 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:39,339 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d96e2308e3ac4ef09429a07d2ae820b1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d96e2308e3ac4ef09429a07d2ae820b1 2023-04-23 05:56:39,344 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=876, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/ec059a07b3fa4ae0b4761208eca0dfbd 2023-04-23 05:56:39,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:39,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:39,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 515 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229459415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55868 deadline: 1682229459416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229459416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229459417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229459419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 517 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229459520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55868 deadline: 1682229459520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229459520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229459523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,660 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/fb77f66cb9fe4b2ebb07caf860125c11 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/fb77f66cb9fe4b2ebb07caf860125c11 2023-04-23 05:56:39,665 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/B of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into fb77f66cb9fe4b2ebb07caf860125c11(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:39,665 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:39,665 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, priority=11, startTime=1682229399227; duration=0sec 2023-04-23 05:56:39,665 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:39,708 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/5b101c5cf8aa4bc2b997f5430905db35 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/5b101c5cf8aa4bc2b997f5430905db35 2023-04-23 05:56:39,712 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/C of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 5b101c5cf8aa4bc2b997f5430905db35(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:39,712 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:39,712 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, priority=11, startTime=1682229399227; duration=0sec 2023-04-23 05:56:39,712 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:39,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 519 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229459724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55868 deadline: 1682229459724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229459724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:39,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229459726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,754 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=876 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/c123fd70b0b54aa3b99b43ec22182237 2023-04-23 05:56:39,764 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=876 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/fc7e1b0d92b046248214a9cee46c3b8e 2023-04-23 05:56:39,768 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/ec059a07b3fa4ae0b4761208eca0dfbd as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ec059a07b3fa4ae0b4761208eca0dfbd 2023-04-23 05:56:39,773 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ec059a07b3fa4ae0b4761208eca0dfbd, entries=150, sequenceid=876, filesize=25.4 K 2023-04-23 05:56:39,774 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/c123fd70b0b54aa3b99b43ec22182237 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c123fd70b0b54aa3b99b43ec22182237 2023-04-23 05:56:39,777 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c123fd70b0b54aa3b99b43ec22182237, entries=150, sequenceid=876, filesize=11.9 K 2023-04-23 05:56:39,777 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/fc7e1b0d92b046248214a9cee46c3b8e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fc7e1b0d92b046248214a9cee46c3b8e 2023-04-23 05:56:39,784 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fc7e1b0d92b046248214a9cee46c3b8e, entries=150, sequenceid=876, filesize=11.9 K 2023-04-23 05:56:39,785 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=46.96 KB/48090 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 465ms, sequenceid=876, compaction requested=false 2023-04-23 05:56:39,785 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:39,785 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,785 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:39,785 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:39,785 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,785 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:39,785 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:39,788 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,788 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,789 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,789 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:39,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a6e9fe[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:39,791 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:39,791 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,792 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,792 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:39,792 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:39,792 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:39,792 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,792 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:39,793 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:39,793 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:39,793 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,793 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:39,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:39,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:39,797 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:39,797 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:39,797 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:39,797 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:39,797 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:39,799 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,799 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,799 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,799 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,801 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,805 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:39,805 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:39,805 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:39,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:39,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:39,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:39,805 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:39,805 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:39,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:39,805 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,805 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,805 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,806 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,806 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,806 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:39,806 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:39,906 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:39,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:39,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:39,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:39,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:39,909 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:39,909 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:39,909 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:39,909 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:39,910 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,910 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:39,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:39,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,911 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:39,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:39,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:39,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:39,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:39,912 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:39,913 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:39,913 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:39,913 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:39,913 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:39,913 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,913 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:39,913 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-23 05:56:39,913 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:39,913 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:39,913 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:39,913 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:39,914 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:39,914 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:39,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:39,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:39,927 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042395df11ba1c70417e972263d087959f8b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042395df11ba1c70417e972263d087959f8b 2023-04-23 05:56:39,950 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=888, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/c4557aed455645749420cfe7533e5601 2023-04-23 05:56:39,959 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/aac1676d3b8e44a9b2074432a186680e 2023-04-23 05:56:39,967 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/b8554ec7e02e4d70abf07ff977b19891 2023-04-23 05:56:39,971 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/c4557aed455645749420cfe7533e5601 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c4557aed455645749420cfe7533e5601 2023-04-23 05:56:39,975 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c4557aed455645749420cfe7533e5601, entries=150, sequenceid=888, filesize=25.4 K 2023-04-23 05:56:39,975 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/aac1676d3b8e44a9b2074432a186680e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aac1676d3b8e44a9b2074432a186680e 2023-04-23 05:56:39,979 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aac1676d3b8e44a9b2074432a186680e, entries=150, sequenceid=888, filesize=11.9 K 2023-04-23 05:56:39,979 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/b8554ec7e02e4d70abf07ff977b19891 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b8554ec7e02e4d70abf07ff977b19891 2023-04-23 05:56:39,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229399966, max timestamp=1682229399979Num uniques 859; 2023-04-23 05:56:39,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229399966, max timestamp=1682229399979Num uniques 859; 2023-04-23 05:56:39,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229399966, max timestamp=1682229399979Num uniques 859; 2023-04-23 05:56:39,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:39,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:39,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:39,985 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b8554ec7e02e4d70abf07ff977b19891, entries=150, sequenceid=888, filesize=11.9 K 2023-04-23 05:56:39,985 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=33.54 KB/34350 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 72ms, sequenceid=888, compaction requested=true 2023-04-23 05:56:39,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:39,985 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:39,986 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,986 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:39,986 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:39,987 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,987 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:39,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70a339c0[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:39,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:39,989 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,990 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:39,990 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,991 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:39,990 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:39,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:39,991 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:39,993 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:39,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,993 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:39,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:39,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-23 05:56:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:39,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:39,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:39,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:39,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:39,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:39,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:39,996 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:39,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:39,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:39,996 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:39,996 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:39,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:39,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,998 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:39,999 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,000 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,002 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:40,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:40,002 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,003 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:40,003 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:40,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:40,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,016 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423d86eba10df0143c3a8fd565a1bf44eb7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d86eba10df0143c3a8fd565a1bf44eb7 2023-04-23 05:56:40,021 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=913, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/aad72943a89542d0af909680fa467d47 2023-04-23 05:56:40,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229400034, max timestamp=1682229400040Num uniques 859; 2023-04-23 05:56:40,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682229400034, max timestamp=1682229400040Num uniques 859; 2023-04-23 05:56:40,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:40,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-23 05:56:40,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682229400034, max timestamp=1682229400040Num uniques 859; 2023-04-23 05:56:40,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-23 05:56:40,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=913 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/e16132aec7434407a3c7206019be2548 2023-04-23 05:56:40,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=913 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/e446a50cbd5c4754b644e94d715c6e61 2023-04-23 05:56:40,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 525 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229460075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229460076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682229400066, max timestamp=1682229400078Num uniques 1018; 2023-04-23 05:56:40,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-23 05:56:40,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682229400066, max timestamp=1682229400078Num uniques 1009; 2023-04-23 05:56:40,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682229400066, max timestamp=1682229400078Num uniques 1018; 2023-04-23 05:56:40,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-23 05:56:40,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-04-23 05:56:40,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/aad72943a89542d0af909680fa467d47 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/aad72943a89542d0af909680fa467d47 2023-04-23 05:56:40,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/aad72943a89542d0af909680fa467d47, entries=150, sequenceid=913, filesize=25.4 K 2023-04-23 05:56:40,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/e16132aec7434407a3c7206019be2548 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e16132aec7434407a3c7206019be2548 2023-04-23 05:56:40,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e16132aec7434407a3c7206019be2548, entries=150, sequenceid=913, filesize=11.9 K 2023-04-23 05:56:40,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/e446a50cbd5c4754b644e94d715c6e61 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e446a50cbd5c4754b644e94d715c6e61 2023-04-23 05:56:40,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e446a50cbd5c4754b644e94d715c6e61, entries=150, sequenceid=913, filesize=11.9 K 2023-04-23 05:56:40,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=87.22 KB/89310 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 100ms, sequenceid=913, compaction requested=true 2023-04-23 05:56:40,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:40,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:40,094 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:40,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:40,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:40,094 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:40,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:40,095 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=93.93 KB heapSize=255.23 KB 2023-04-23 05:56:40,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:40,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:40,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:40,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:40,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:40,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-23 05:56:40,095 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105172 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:40,095 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/A is initiating minor compaction (all files) 2023-04-23 05:56:40,095 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/A in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,096 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cb828972c2624bf4b8ae47a0bf76388e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ec059a07b3fa4ae0b4761208eca0dfbd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c4557aed455645749420cfe7533e5601, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/aad72943a89542d0af909680fa467d47] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=102.7 K 2023-04-23 05:56:40,103 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:40,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:40,104 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:40,104 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/B is initiating minor compaction (all files) 2023-04-23 05:56:40,104 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting cb828972c2624bf4b8ae47a0bf76388e, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1682229398748 2023-04-23 05:56:40,104 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/B in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,105 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/fb77f66cb9fe4b2ebb07caf860125c11, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c123fd70b0b54aa3b99b43ec22182237, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aac1676d3b8e44a9b2074432a186680e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e16132aec7434407a3c7206019be2548] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=48.6 K 2023-04-23 05:56:40,105 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting ec059a07b3fa4ae0b4761208eca0dfbd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=876, earliestPutTs=1682229398788 2023-04-23 05:56:40,105 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting fb77f66cb9fe4b2ebb07caf860125c11, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1682229398748 2023-04-23 05:56:40,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:40,105 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c123fd70b0b54aa3b99b43ec22182237, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=876, earliestPutTs=1682229398788 2023-04-23 05:56:40,106 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting c4557aed455645749420cfe7533e5601, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1682229399415 2023-04-23 05:56:40,107 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting aad72943a89542d0af909680fa467d47, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=913, earliestPutTs=1682229399976 2023-04-23 05:56:40,107 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting aac1676d3b8e44a9b2074432a186680e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1682229399415 2023-04-23 05:56:40,107 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e16132aec7434407a3c7206019be2548, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=913, earliestPutTs=1682229399976 2023-04-23 05:56:40,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:40,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:40,112 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:40,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:40,112 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:40,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,113 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,114 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,114 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,115 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:40,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:40,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:40,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:40,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,116 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:40,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229460120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55868 deadline: 1682229460120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229460121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,126 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#B#compaction#605 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:40,140 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:40,140 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,140 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:40,140 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:40,140 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,141 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:40,141 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:40,141 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:40,144 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:40,147 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,147 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,148 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#A#compaction#606 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:40,148 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,148 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:40,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c66f1c1[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:40,151 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:40,151 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,155 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:40,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,155 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:40,157 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,158 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:40,158 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,158 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:40,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:40,160 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:40,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:40,160 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:40,161 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:40,161 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:40,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,163 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,164 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,165 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,169 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,169 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,169 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,169 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:40,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:40,169 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,169 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,170 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,170 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,170 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,170 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,170 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:40,170 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:40,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 527 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229460180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229460181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229460226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55868 deadline: 1682229460226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229460227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,270 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:40,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:40,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:40,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:40,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:40,274 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:40,274 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,274 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:40,274 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:40,274 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,274 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,275 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,275 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,276 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:40,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:40,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:40,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:40,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:40,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:40,277 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:40,278 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,278 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:40,278 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:40,278 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,278 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:40,278 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:40,278 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,278 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:40,278 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:40,280 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,280 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,280 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,280 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:40,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24987192[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:40,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:40,282 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,283 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,283 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,283 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:40,283 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:40,283 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:40,284 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:40,286 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,286 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:40,286 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,286 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:40,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:40,288 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:40,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:40,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:40,288 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:40,288 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:40,290 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,290 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,290 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,290 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,291 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,293 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:40,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:40,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,293 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,294 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:40,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:40,294 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:40,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 529 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229460385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229460385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,394 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:40,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:40,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:40,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:40,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:40,397 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:40,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:40,397 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:40,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,398 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,399 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,399 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,399 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,399 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:40,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:40,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:40,400 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:40,401 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,401 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:40,401 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:40,401 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:40,403 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,403 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,404 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:40,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@316495e6[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:40,405 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:40,406 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,406 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:40,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:40,407 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:40,407 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,407 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:40,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:40,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,408 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,408 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:40,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:40,411 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:40,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:40,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:40,411 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:40,411 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:40,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,412 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,413 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,414 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,418 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:40,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:40,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,418 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:40,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,418 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:40,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,418 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229460428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55868 deadline: 1682229460429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229460429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,518 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:40,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:40,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:40,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:40,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:40,522 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:40,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:40,522 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:40,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,523 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,524 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,524 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:40,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:40,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:40,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:40,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,526 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:40,527 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,527 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:40,527 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:40,527 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:40,529 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,529 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,529 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,529 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:40,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5860d733[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:40,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:40,530 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,531 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,531 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:40,531 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:40,531 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:40,531 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,532 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:40,536 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,536 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,536 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,537 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:40,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:40,539 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:40,539 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:40,539 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:40,539 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:40,539 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:40,540 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,540 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,540 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,541 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,543 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,545 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,545 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:40,545 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:40,545 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:40,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,545 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,546 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,546 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,546 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:40,546 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:40,546 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,546 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,546 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,553 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/597848dc40704995a5a3d8ffa075b0d2 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/597848dc40704995a5a3d8ffa075b0d2 2023-04-23 05:56:40,555 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304239837e9a33d414d8bbfda4f29573d0707 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304239837e9a33d414d8bbfda4f29573d0707 2023-04-23 05:56:40,566 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=959, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/4913bf1a843d49e39f1d337ae6d96c7f 2023-04-23 05:56:40,566 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/B of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 597848dc40704995a5a3d8ffa075b0d2(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:40,567 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:40,567 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, priority=12, startTime=1682229400094; duration=0sec 2023-04-23 05:56:40,567 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:40,567 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:40,569 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:40,569 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/C is initiating minor compaction (all files) 2023-04-23 05:56:40,570 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/C in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,570 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/5b101c5cf8aa4bc2b997f5430905db35, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fc7e1b0d92b046248214a9cee46c3b8e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b8554ec7e02e4d70abf07ff977b19891, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e446a50cbd5c4754b644e94d715c6e61] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=48.5 K 2023-04-23 05:56:40,571 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 5b101c5cf8aa4bc2b997f5430905db35, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1682229398748 2023-04-23 05:56:40,572 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting fc7e1b0d92b046248214a9cee46c3b8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=876, earliestPutTs=1682229398788 2023-04-23 05:56:40,572 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b8554ec7e02e4d70abf07ff977b19891, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1682229399415 2023-04-23 05:56:40,573 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting e446a50cbd5c4754b644e94d715c6e61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=913, earliestPutTs=1682229399976 2023-04-23 05:56:40,576 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/7c4f2027e6734dbf96677a3b71b3e81c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/7c4f2027e6734dbf96677a3b71b3e81c 2023-04-23 05:56:40,581 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/A of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 7c4f2027e6734dbf96677a3b71b3e81c(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:40,581 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:40,581 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, priority=12, startTime=1682229400094; duration=0sec 2023-04-23 05:56:40,581 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:40,585 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#C#compaction#609 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:40,646 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:40,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:40,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:40,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:40,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:40,649 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:40,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:40,649 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:40,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,650 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,653 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,653 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,654 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:40,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:40,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:40,655 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:40,655 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,655 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:40,655 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:40,655 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:40,656 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,656 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:40,656 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:40,656 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:40,658 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,658 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,658 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,658 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:40,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@353dd07[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:40,659 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:40,659 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,660 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,661 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:40,661 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:40,661 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:40,666 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,666 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:40,667 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:40,667 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,667 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,667 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:40,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:40,669 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:40,669 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:40,669 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:40,669 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:40,669 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:40,672 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,672 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,672 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,673 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,674 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:40,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:40,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:40,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,676 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,676 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,677 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:40,677 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,677 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:40,677 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,677 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 531 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229460690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229460692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229460731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55868 deadline: 1682229460733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:40,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229460733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,777 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:40,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:40,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:40,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:40,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:40,780 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:40,780 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,780 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:40,780 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:40,781 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,781 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,782 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,782 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:40,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:40,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:40,783 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:40,783 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,783 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:40,784 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,784 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:40,784 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:40,784 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:40,786 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,786 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,786 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,786 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:40,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6420281e[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:40,787 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:40,787 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,789 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,789 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:40,789 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:40,789 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:40,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,790 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:40,791 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,791 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:40,791 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,791 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:40,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:40,793 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:40,793 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:40,793 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:40,794 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:40,794 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:40,795 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,795 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,795 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,796 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,797 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,800 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:40,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:40,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:40,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,800 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,800 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:40,800 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:40,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,901 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:40,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:40,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:40,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:40,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:40,904 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:40,904 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,904 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:40,904 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:40,905 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,905 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,907 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,907 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,908 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,908 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:40,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:40,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:40,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:40,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:40,909 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,909 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:40,909 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:40,909 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:40,909 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:40,911 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,911 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,911 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,911 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:40,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:40,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@717e4ad9[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:40,913 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:40,913 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,914 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,914 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:40,914 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:40,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,914 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:40,914 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:40,915 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:40,915 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,915 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:40,916 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:40,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:40,918 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:40,918 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:40,918 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:40,918 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:40,918 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:40,920 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:40,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,921 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:40,921 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,923 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:40,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:40,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:40,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:40,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:40,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:40,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:40,928 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:40,928 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:40,928 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:40,928 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:40,929 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:40,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:40,985 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=959 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/c042e6487c0040d7a1321ec7a12c87f5 2023-04-23 05:56:40,995 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/d523fca8f9994b2986b87d7522179433 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d523fca8f9994b2986b87d7522179433 2023-04-23 05:56:40,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=959 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/d9de3f05dd6f4178a8cafd728166a417 2023-04-23 05:56:41,000 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/C of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into d523fca8f9994b2986b87d7522179433(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:41,001 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:41,001 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, priority=12, startTime=1682229400094; duration=0sec 2023-04-23 05:56:41,001 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:41,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/4913bf1a843d49e39f1d337ae6d96c7f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/4913bf1a843d49e39f1d337ae6d96c7f 2023-04-23 05:56:41,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/4913bf1a843d49e39f1d337ae6d96c7f, entries=250, sequenceid=959, filesize=39.2 K 2023-04-23 05:56:41,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/c042e6487c0040d7a1321ec7a12c87f5 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c042e6487c0040d7a1321ec7a12c87f5 2023-04-23 05:56:41,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c042e6487c0040d7a1321ec7a12c87f5, entries=150, sequenceid=959, filesize=11.9 K 2023-04-23 05:56:41,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/d9de3f05dd6f4178a8cafd728166a417 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d9de3f05dd6f4178a8cafd728166a417 2023-04-23 05:56:41,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d9de3f05dd6f4178a8cafd728166a417, entries=150, sequenceid=959, filesize=11.9 K 2023-04-23 05:56:41,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~255.19 KB/261312, currentSize=87.22 KB/89310 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 924ms, sequenceid=959, compaction requested=false 2023-04-23 05:56:41,018 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:41,029 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:41,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:41,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:41,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:41,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:41,032 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:41,032 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:41,032 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:41,032 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:41,033 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,033 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,034 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:41,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:41,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,034 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:41,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:41,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:41,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:41,035 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:41,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:41,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:41,036 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:41,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:41,036 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-23 05:56:41,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:41,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:41,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:41,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:41,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:41,036 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:41,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:41,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:41,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 540 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229461215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229461215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40995] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55894 deadline: 1682229461236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55868 deadline: 1682229461237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55906 deadline: 1682229461239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 542 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229461320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229461320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,461 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423f5c43122db964454b51f40ebe4fd9ba3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f5c43122db964454b51f40ebe4fd9ba3 2023-04-23 05:56:41,466 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=978, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e64b589ab3d84135830f18cf87c18e19 2023-04-23 05:56:41,475 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/1f8f56a144124fe98ade943d92922299 2023-04-23 05:56:41,485 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/b0a1f5bb61aa471fa225b61c63ff1d0e 2023-04-23 05:56:41,489 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/e64b589ab3d84135830f18cf87c18e19 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e64b589ab3d84135830f18cf87c18e19 2023-04-23 05:56:41,492 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e64b589ab3d84135830f18cf87c18e19, entries=150, sequenceid=978, filesize=25.4 K 2023-04-23 05:56:41,493 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/1f8f56a144124fe98ade943d92922299 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1f8f56a144124fe98ade943d92922299 2023-04-23 05:56:41,497 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1f8f56a144124fe98ade943d92922299, entries=150, sequenceid=978, filesize=11.9 K 2023-04-23 05:56:41,497 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/b0a1f5bb61aa471fa225b61c63ff1d0e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b0a1f5bb61aa471fa225b61c63ff1d0e 2023-04-23 05:56:41,501 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b0a1f5bb61aa471fa225b61c63ff1d0e, entries=150, sequenceid=978, filesize=11.9 K 2023-04-23 05:56:41,502 INFO [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 466ms, sequenceid=978, compaction requested=true 2023-04-23 05:56:41,502 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:41,502 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:41,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:41,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:41,502 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:41,504 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,504 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,504 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,504 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:41,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f79c28a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:41,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:41,506 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:41,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:41,507 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:41,507 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,507 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:41,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:41,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:41,508 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:41,508 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:41,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:41,511 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:41,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:41,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:41,511 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:41,511 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:41,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,514 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:41,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:41,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:41,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,515 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,516 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:41,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:41,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:41,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:41,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:41,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:41,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,519 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:41,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:41,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,519 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,519 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:41,520 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:41,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:41,524 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-23 05:56:41,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:41,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:41,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:41,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:41,525 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:41,525 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:41,542 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304236684d70bea304e319299eea39fe0d639 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236684d70bea304e319299eea39fe0d639 2023-04-23 05:56:41,547 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=996, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/05021aaa07334731adf12f2b9d22148e 2023-04-23 05:56:41,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229461553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40995] ipc.CallRunner(144): callId: 551 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229461553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,620 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:41,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:41,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:41,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:41,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:41,623 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:41,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:41,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:41,623 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:41,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,624 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,625 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:41,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:41,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:41,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,625 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:41,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:41,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:41,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:41,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:41,626 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:41,626 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:41,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:41,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:41,627 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:41,629 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,629 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:41,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@325f9987[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:41,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:41,631 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,633 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:41,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:41,633 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:41,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,633 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:41,634 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:41,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,634 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:41,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:41,638 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:41,638 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:41,638 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:41,638 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:41,638 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:41,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,639 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:41,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:41,640 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,641 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:41,643 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:41,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:41,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:41,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:41,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,643 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:41,644 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:41,644 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:41,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:41,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:41,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229461658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 553 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229461658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,744 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:41,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:41,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:41,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:41,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:41,747 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:41,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:41,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:41,747 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:41,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,748 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,750 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:41,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,750 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:41,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:41,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:41,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:41,751 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:41,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:41,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:41,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:41,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:41,752 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:41,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:41,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:41,752 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:41,754 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,754 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:41,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@575a370a[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:41,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:41,756 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,757 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,757 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:41,757 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:41,759 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:41,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:41,759 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:41,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:41,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:41,763 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:41,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:41,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:41,763 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:41,763 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,766 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:41,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,767 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,768 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,776 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:41,776 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:41,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:41,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:41,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-23 05:56:41,776 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,776 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,776 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,776 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,777 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,777 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,776 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:41,776 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:41,776 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:41,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:41,777 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:41,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 555 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55900 deadline: 1682229461862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-23 05:56:41,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:55936 deadline: 1682229461862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,877 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:41,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:41,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-23 05:56:41,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-23 05:56:41,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-23 05:56:41,881 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-23 05:56:41,881 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:41,881 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-23 05:56:41,881 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-23 05:56:41,881 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,881 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,882 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:41,882 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:41,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,883 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-23 05:56:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-23 05:56:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-23 05:56:41,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-23 05:56:41,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-23 05:56:41,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-23 05:56:41,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-23 05:56:41,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-23 05:56:41,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-23 05:56:41,884 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-23 05:56:41,884 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,884 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. started... 2023-04-23 05:56:41,884 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. as already flushing 2023-04-23 05:56:41,884 DEBUG [rs(jenkins-hbase4.apache.org,40995,1682229223026)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-23 05:56:41,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-23 05:56:41,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-23 05:56:41,885 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-23 05:56:41,887 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,887 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,888 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-23 05:56:41,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,40995,1682229223026' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-23 05:56:41,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547187ba[Count = 0] remaining members to acquire global barrier 2023-04-23 05:56:41,889 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-23 05:56:41,889 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,890 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,890 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-23 05:56:41,891 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-23 05:56:41,891 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,40995,1682229223026' in zk 2023-04-23 05:56:41,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,891 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-23 05:56:41,892 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-23 05:56:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,892 DEBUG [member: 'jenkins-hbase4.apache.org,40995,1682229223026' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-23 05:56:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,40995,1682229223026': 2023-04-23 05:56:41,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,40995,1682229223026' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-23 05:56:41,895 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-23 05:56:41,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-23 05:56:41,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-23 05:56:41,895 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-23 05:56:41,895 INFO [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-23 05:56:41,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-23 05:56:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-23 05:56:41,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,896 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:41,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-23 05:56:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,897 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-23 05:56:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-23 05:56:41,899 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-23 05:56:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,902 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,902 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,902 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-23 05:56:41,902 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-23 05:56:41,902 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-23 05:56:41,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-23 05:56:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-23 05:56:41,903 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:41,903 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,903 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-23 05:56:41,903 DEBUG [(jenkins-hbase4.apache.org,44901,1682229221800)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-23 05:56:41,903 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-23 05:56:41,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-23 05:56:41,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-23 05:56:41,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-23 05:56:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-23 05:56:41,903 DEBUG [Thread-3847] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-23 05:56:41,903 DEBUG [Thread-3847] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-23 05:56:41,905 DEBUG [Thread-3856] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x430757a6 to 127.0.0.1:63802 2023-04-23 05:56:41,905 DEBUG [Thread-3856] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:41,906 DEBUG [Thread-3852] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5fdfef3b to 127.0.0.1:63802 2023-04-23 05:56:41,907 DEBUG [Thread-3852] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:41,907 DEBUG [Thread-3850] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x785e366b to 127.0.0.1:63802 2023-04-23 05:56:41,907 DEBUG [Thread-3850] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:41,907 DEBUG [Thread-3854] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77f9399b to 127.0.0.1:63802 2023-04-23 05:56:41,909 DEBUG [Thread-3854] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:41,911 DEBUG [Thread-3848] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x156e4ba6 to 127.0.0.1:63802 2023-04-23 05:56:41,911 DEBUG [Thread-3848] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:41,956 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=996 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/d47cdd06000f48bca53247ddc7d2e17c 2023-04-23 05:56:41,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=996 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/030874d814c54be8b4182a0c53e43ab3 2023-04-23 05:56:41,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/05021aaa07334731adf12f2b9d22148e as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/05021aaa07334731adf12f2b9d22148e 2023-04-23 05:56:41,969 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/05021aaa07334731adf12f2b9d22148e, entries=200, sequenceid=996, filesize=32.3 K 2023-04-23 05:56:41,970 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/d47cdd06000f48bca53247ddc7d2e17c as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d47cdd06000f48bca53247ddc7d2e17c 2023-04-23 05:56:41,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d47cdd06000f48bca53247ddc7d2e17c, entries=150, sequenceid=996, filesize=11.9 K 2023-04-23 05:56:41,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/030874d814c54be8b4182a0c53e43ab3 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/030874d814c54be8b4182a0c53e43ab3 2023-04-23 05:56:41,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/030874d814c54be8b4182a0c53e43ab3, entries=150, sequenceid=996, filesize=11.9 K 2023-04-23 05:56:41,976 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 452ms, sequenceid=996, compaction requested=true 2023-04-23 05:56:41,976 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:41,976 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:41,976 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:41,977 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:41,977 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:41,977 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:41,978 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 126432 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:41,978 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:41,978 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/A is initiating minor compaction (all files) 2023-04-23 05:56:41,978 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/B is initiating minor compaction (all files) 2023-04-23 05:56:41,978 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/A in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,978 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/B in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,978 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/7c4f2027e6734dbf96677a3b71b3e81c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/4913bf1a843d49e39f1d337ae6d96c7f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e64b589ab3d84135830f18cf87c18e19, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/05021aaa07334731adf12f2b9d22148e] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=123.5 K 2023-04-23 05:56:41,978 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/597848dc40704995a5a3d8ffa075b0d2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c042e6487c0040d7a1321ec7a12c87f5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1f8f56a144124fe98ade943d92922299, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d47cdd06000f48bca53247ddc7d2e17c] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=48.7 K 2023-04-23 05:56:41,978 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 7c4f2027e6734dbf96677a3b71b3e81c, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=913, earliestPutTs=1682229399976 2023-04-23 05:56:41,978 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 597848dc40704995a5a3d8ffa075b0d2, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=913, earliestPutTs=1682229399976 2023-04-23 05:56:41,978 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 4913bf1a843d49e39f1d337ae6d96c7f, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=959, earliestPutTs=1682229400082 2023-04-23 05:56:41,978 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting c042e6487c0040d7a1321ec7a12c87f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=959, earliestPutTs=1682229400087 2023-04-23 05:56:41,979 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting e64b589ab3d84135830f18cf87c18e19, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1682229400119 2023-04-23 05:56:41,979 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 1f8f56a144124fe98ade943d92922299, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1682229400119 2023-04-23 05:56:41,979 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] compactions.Compactor(207): Compacting 05021aaa07334731adf12f2b9d22148e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=996, earliestPutTs=1682229401205 2023-04-23 05:56:41,979 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d47cdd06000f48bca53247ddc7d2e17c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=996, earliestPutTs=1682229401205 2023-04-23 05:56:41,985 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#B#compaction#618 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:41,988 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#A#compaction#617 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:41,992 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/71a7b132d6524799bec41a64d0da7e6a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/71a7b132d6524799bec41a64d0da7e6a 2023-04-23 05:56:41,995 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/B of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 71a7b132d6524799bec41a64d0da7e6a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:41,995 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:41,995 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/B, priority=12, startTime=1682229401976; duration=0sec 2023-04-23 05:56:41,996 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-23 05:56:41,996 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-23 05:56:41,997 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-23 05:56:41,997 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1912): 9ad1ed5a4f4248a6e5f0e43c7f776f48/C is initiating minor compaction (all files) 2023-04-23 05:56:41,997 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9ad1ed5a4f4248a6e5f0e43c7f776f48/C in TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:41,997 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d523fca8f9994b2986b87d7522179433, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d9de3f05dd6f4178a8cafd728166a417, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b0a1f5bb61aa471fa225b61c63ff1d0e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/030874d814c54be8b4182a0c53e43ab3] into tmpdir=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp, totalSize=48.7 K 2023-04-23 05:56:41,997 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d523fca8f9994b2986b87d7522179433, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=913, earliestPutTs=1682229399976 2023-04-23 05:56:41,997 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting d9de3f05dd6f4178a8cafd728166a417, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=959, earliestPutTs=1682229400087 2023-04-23 05:56:41,998 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting b0a1f5bb61aa471fa225b61c63ff1d0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=978, earliestPutTs=1682229400119 2023-04-23 05:56:41,998 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] compactions.Compactor(207): Compacting 030874d814c54be8b4182a0c53e43ab3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=996, earliestPutTs=1682229401205 2023-04-23 05:56:42,000 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/72882dad7bc447eca9889e639b2719ef as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/72882dad7bc447eca9889e639b2719ef 2023-04-23 05:56:42,003 DEBUG [Thread-3847] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-23 05:56:42,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-23 05:56:42,004 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] throttle.PressureAwareThroughputController(145): 9ad1ed5a4f4248a6e5f0e43c7f776f48#C#compaction#619 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-23 05:56:42,004 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/A of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 72882dad7bc447eca9889e639b2719ef(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:42,004 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:42,004 INFO [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/A, priority=12, startTime=1682229401976; duration=0sec 2023-04-23 05:56:42,004 DEBUG [RS:0;jenkins-hbase4:40995-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:42,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40995] regionserver.HRegion(9158): Flush requested on 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:42,168 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-23 05:56:42,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:42,168 DEBUG [Thread-3837] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a320817 to 127.0.0.1:63802 2023-04-23 05:56:42,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:42,168 DEBUG [Thread-3837] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:42,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:42,168 DEBUG [Thread-3839] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19c1e3ee to 127.0.0.1:63802 2023-04-23 05:56:42,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:42,168 DEBUG [Thread-3839] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:42,168 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:42,168 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:42,244 DEBUG [Thread-3845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3420e54c to 127.0.0.1:63802 2023-04-23 05:56:42,244 DEBUG [Thread-3841] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x526f2221 to 127.0.0.1:63802 2023-04-23 05:56:42,244 DEBUG [Thread-3845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:42,244 DEBUG [Thread-3841] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:42,247 DEBUG [Thread-3843] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17cf2e3f to 127.0.0.1:63802 2023-04-23 05:56:42,247 DEBUG [Thread-3843] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 238 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 414 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 4556 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 4468 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 4439 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 4607 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(398): read 4463 2023-04-23 05:56:42,248 INFO [Listener at localhost/41845] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-23 05:56:42,249 INFO [Listener at localhost/41845] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-23 05:56:42,249 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52104714 to 127.0.0.1:63802 2023-04-23 05:56:42,249 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:42,249 INFO [Listener at localhost/41845] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-23 05:56:42,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-23 05:56:42,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:42,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-23 05:56:42,253 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229402253"}]},"ts":"1682229402253"} 2023-04-23 05:56:42,253 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-23 05:56:42,256 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-23 05:56:42,256 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, UNASSIGN}] 2023-04-23 05:56:42,258 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, UNASSIGN 2023-04-23 05:56:42,258 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=9ad1ed5a4f4248a6e5f0e43c7f776f48, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:42,259 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229402258"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682229402258"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682229402258"}]},"ts":"1682229402258"} 2023-04-23 05:56:42,259 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026}] 2023-04-23 05:56:42,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-23 05:56:42,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:42,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9ad1ed5a4f4248a6e5f0e43c7f776f48, disabling compactions & flushes 2023-04-23 05:56:42,412 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:42,413 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/259fe19f5e344db9b136495e6e35876f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/259fe19f5e344db9b136495e6e35876f 2023-04-23 05:56:42,416 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9ad1ed5a4f4248a6e5f0e43c7f776f48/C of 9ad1ed5a4f4248a6e5f0e43c7f776f48 into 259fe19f5e344db9b136495e6e35876f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-23 05:56:42,416 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:42,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:42,416 INFO [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48., storeName=9ad1ed5a4f4248a6e5f0e43c7f776f48/C, priority=12, startTime=1682229401977; duration=0sec 2023-04-23 05:56:42,416 DEBUG [RS:0;jenkins-hbase4:40995-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-23 05:56:42,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-23 05:56:42,581 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230423dd6d75715f8e4cb98c5b173f74760b93 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423dd6d75715f8e4cb98c5b173f74760b93 2023-04-23 05:56:42,585 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1015, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/b8e8b75bce4c4da59e47553890bb4113 2023-04-23 05:56:42,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-23 05:56:42,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1015 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/032fc9136dcd448ca4406369b314c209 2023-04-23 05:56:43,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-23 05:56:43,406 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1015 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/4ba33c10dfed456aad98ddee2b3ca960 2023-04-23 05:56:43,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/b8e8b75bce4c4da59e47553890bb4113 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/b8e8b75bce4c4da59e47553890bb4113 2023-04-23 05:56:43,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/b8e8b75bce4c4da59e47553890bb4113, entries=150, sequenceid=1015, filesize=25.4 K 2023-04-23 05:56:43,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/032fc9136dcd448ca4406369b314c209 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/032fc9136dcd448ca4406369b314c209 2023-04-23 05:56:43,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/032fc9136dcd448ca4406369b314c209, entries=150, sequenceid=1015, filesize=11.9 K 2023-04-23 05:56:43,417 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/4ba33c10dfed456aad98ddee2b3ca960 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/4ba33c10dfed456aad98ddee2b3ca960 2023-04-23 05:56:43,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/4ba33c10dfed456aad98ddee2b3ca960, entries=150, sequenceid=1015, filesize=11.9 K 2023-04-23 05:56:43,420 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=20.13 KB/20610 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 1252ms, sequenceid=1015, compaction requested=false 2023-04-23 05:56:43,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:43,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:43,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:43,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. after waiting 0 ms 2023-04-23 05:56:43,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:43,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9ad1ed5a4f4248a6e5f0e43c7f776f48 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-23 05:56:43,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=A 2023-04-23 05:56:43,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:43,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=B 2023-04-23 05:56:43,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:43,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9ad1ed5a4f4248a6e5f0e43c7f776f48, store=C 2023-04-23 05:56:43,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-23 05:56:43,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304230950098d9e47453fa4ede6bea90798bf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304230950098d9e47453fa4ede6bea90798bf 2023-04-23 05:56:43,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1022, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/de37c0980dff4dc093e95b191fc1b376 2023-04-23 05:56:43,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1022 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/1e66424c976a479cbcda98fb9facd245 2023-04-23 05:56:44,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1022 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/6d3375b9b09543b393a65883e782a70a 2023-04-23 05:56:44,252 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/A/de37c0980dff4dc093e95b191fc1b376 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/de37c0980dff4dc093e95b191fc1b376 2023-04-23 05:56:44,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/de37c0980dff4dc093e95b191fc1b376, entries=50, sequenceid=1022, filesize=11.7 K 2023-04-23 05:56:44,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/B/1e66424c976a479cbcda98fb9facd245 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1e66424c976a479cbcda98fb9facd245 2023-04-23 05:56:44,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1e66424c976a479cbcda98fb9facd245, entries=50, sequenceid=1022, filesize=7.1 K 2023-04-23 05:56:44,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/.tmp/C/6d3375b9b09543b393a65883e782a70a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/6d3375b9b09543b393a65883e782a70a 2023-04-23 05:56:44,262 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/6d3375b9b09543b393a65883e782a70a, entries=50, sequenceid=1022, filesize=7.1 K 2023-04-23 05:56:44,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 9ad1ed5a4f4248a6e5f0e43c7f776f48 in 843ms, sequenceid=1022, compaction requested=true 2023-04-23 05:56:44,264 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e132fc7418c341a19941f700eb2a1aa0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/666ed33d75b14d6f9cdc31e651597981, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/0f73c4baabac4c96a53909ccd13f612b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/074fbcaab1e440ccba815a93aac89059, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c3fee99ca6544ca59713813832ee919f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1210494ebb2949d0afca4cd5ab935c28, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/918f74a83f644d0f80310d74a06f00a9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/20b0670e815d4f9b91f0928fce367b2c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/8ea3db4ce1b4469ba42cdcb768816843, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1c35c1e6ced142df840d15361759d768, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/5b465749d5db4ba080d1cf6706e65b6d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bc52048609bf4a13a501a4bc1ab7e367, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bfce5daf612f48b8bea76096f724978b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/d5511574cde141afb850986b3c4ff5bd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cde241e8a5e94625bd1d95c65b7b373b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/261d3bf6299347b49d55479a470fc4af, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/af43f4ab11624195aed3f3c295967593, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c77bb5709f3f44b59af173a7fba9c09a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ecb8d6497b2c455ba907c90e67723a7b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/47f49d9193b54a32b34c3250202226c4, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/668085a0c84f487dae6c6d2cfdebf330, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e320934b8fa54e3eaa3f19f4f66fc606, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e7af765886eb4c3285587aec8fe07b00, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/848e50e7b159413683f0a8e3553d8c6d, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/35dba8b3043940718fb81dc758f9e1f0, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1994d6c3926749e8866c5e78ff1c51ba, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ad3aa4d6ce8646aea81ccc102f787767, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e3ef5bf66f504177b51ab882db1be852, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/9eb6c8d0cbee418d8de8329a9dae926b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cb828972c2624bf4b8ae47a0bf76388e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ec059a07b3fa4ae0b4761208eca0dfbd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c4557aed455645749420cfe7533e5601, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/7c4f2027e6734dbf96677a3b71b3e81c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/aad72943a89542d0af909680fa467d47, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/4913bf1a843d49e39f1d337ae6d96c7f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e64b589ab3d84135830f18cf87c18e19, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/05021aaa07334731adf12f2b9d22148e] to archive 2023-04-23 05:56:44,265 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:56:44,266 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e132fc7418c341a19941f700eb2a1aa0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e132fc7418c341a19941f700eb2a1aa0 2023-04-23 05:56:44,267 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/666ed33d75b14d6f9cdc31e651597981 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/666ed33d75b14d6f9cdc31e651597981 2023-04-23 05:56:44,268 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/0f73c4baabac4c96a53909ccd13f612b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/0f73c4baabac4c96a53909ccd13f612b 2023-04-23 05:56:44,269 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/074fbcaab1e440ccba815a93aac89059 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/074fbcaab1e440ccba815a93aac89059 2023-04-23 05:56:44,270 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c3fee99ca6544ca59713813832ee919f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c3fee99ca6544ca59713813832ee919f 2023-04-23 05:56:44,270 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1210494ebb2949d0afca4cd5ab935c28 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1210494ebb2949d0afca4cd5ab935c28 2023-04-23 05:56:44,271 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/918f74a83f644d0f80310d74a06f00a9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/918f74a83f644d0f80310d74a06f00a9 2023-04-23 05:56:44,272 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/20b0670e815d4f9b91f0928fce367b2c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/20b0670e815d4f9b91f0928fce367b2c 2023-04-23 05:56:44,273 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/8ea3db4ce1b4469ba42cdcb768816843 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/8ea3db4ce1b4469ba42cdcb768816843 2023-04-23 05:56:44,273 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1c35c1e6ced142df840d15361759d768 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1c35c1e6ced142df840d15361759d768 2023-04-23 05:56:44,274 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/5b465749d5db4ba080d1cf6706e65b6d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/5b465749d5db4ba080d1cf6706e65b6d 2023-04-23 05:56:44,275 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bc52048609bf4a13a501a4bc1ab7e367 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bc52048609bf4a13a501a4bc1ab7e367 2023-04-23 05:56:44,276 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bfce5daf612f48b8bea76096f724978b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/bfce5daf612f48b8bea76096f724978b 2023-04-23 05:56:44,276 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/d5511574cde141afb850986b3c4ff5bd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/d5511574cde141afb850986b3c4ff5bd 2023-04-23 05:56:44,277 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cde241e8a5e94625bd1d95c65b7b373b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cde241e8a5e94625bd1d95c65b7b373b 2023-04-23 05:56:44,278 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/261d3bf6299347b49d55479a470fc4af to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/261d3bf6299347b49d55479a470fc4af 2023-04-23 05:56:44,279 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/af43f4ab11624195aed3f3c295967593 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/af43f4ab11624195aed3f3c295967593 2023-04-23 05:56:44,280 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c77bb5709f3f44b59af173a7fba9c09a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c77bb5709f3f44b59af173a7fba9c09a 2023-04-23 05:56:44,280 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ecb8d6497b2c455ba907c90e67723a7b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ecb8d6497b2c455ba907c90e67723a7b 2023-04-23 05:56:44,281 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/47f49d9193b54a32b34c3250202226c4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/47f49d9193b54a32b34c3250202226c4 2023-04-23 05:56:44,282 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/668085a0c84f487dae6c6d2cfdebf330 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/668085a0c84f487dae6c6d2cfdebf330 2023-04-23 05:56:44,283 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e320934b8fa54e3eaa3f19f4f66fc606 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e320934b8fa54e3eaa3f19f4f66fc606 2023-04-23 05:56:44,284 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e7af765886eb4c3285587aec8fe07b00 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e7af765886eb4c3285587aec8fe07b00 2023-04-23 05:56:44,285 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/848e50e7b159413683f0a8e3553d8c6d to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/848e50e7b159413683f0a8e3553d8c6d 2023-04-23 05:56:44,286 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/35dba8b3043940718fb81dc758f9e1f0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/35dba8b3043940718fb81dc758f9e1f0 2023-04-23 05:56:44,287 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1994d6c3926749e8866c5e78ff1c51ba to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/1994d6c3926749e8866c5e78ff1c51ba 2023-04-23 05:56:44,287 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ad3aa4d6ce8646aea81ccc102f787767 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ad3aa4d6ce8646aea81ccc102f787767 2023-04-23 05:56:44,288 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e3ef5bf66f504177b51ab882db1be852 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e3ef5bf66f504177b51ab882db1be852 2023-04-23 05:56:44,289 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/9eb6c8d0cbee418d8de8329a9dae926b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/9eb6c8d0cbee418d8de8329a9dae926b 2023-04-23 05:56:44,290 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cb828972c2624bf4b8ae47a0bf76388e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/cb828972c2624bf4b8ae47a0bf76388e 2023-04-23 05:56:44,291 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ec059a07b3fa4ae0b4761208eca0dfbd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/ec059a07b3fa4ae0b4761208eca0dfbd 2023-04-23 05:56:44,292 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c4557aed455645749420cfe7533e5601 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/c4557aed455645749420cfe7533e5601 2023-04-23 05:56:44,292 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/7c4f2027e6734dbf96677a3b71b3e81c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/7c4f2027e6734dbf96677a3b71b3e81c 2023-04-23 05:56:44,293 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/aad72943a89542d0af909680fa467d47 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/aad72943a89542d0af909680fa467d47 2023-04-23 05:56:44,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/4913bf1a843d49e39f1d337ae6d96c7f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/4913bf1a843d49e39f1d337ae6d96c7f 2023-04-23 05:56:44,295 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e64b589ab3d84135830f18cf87c18e19 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/e64b589ab3d84135830f18cf87c18e19 2023-04-23 05:56:44,296 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/05021aaa07334731adf12f2b9d22148e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/05021aaa07334731adf12f2b9d22148e 2023-04-23 05:56:44,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e48e414f6c274416817ca2d081198e55, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f4fd204e81954f1f88dd62d509898bd3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cd8c8e0871144720b98b118efb30325c, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f841beaeb5f943a99ca59e612411dbcf, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/46cc8b89c9d14031bf63ae031d41db44, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/71f3f79d49324a7c8d2ca26359515087, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cc200a712c564041946e27a674c2e175, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ea918a3838b34b89939f3bb3c2415f89, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/461d5f49a80d4af397555c1aa82aeb4f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/54dd6567b1dc45daa14b24ebd58c2e4f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/6c9f7e74c3c549528d25df6768ee8fb2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/3decae56dcd54246b761fa6d254b5f81, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/07b3d5cff5794f3cbfe0d6ca35de6af3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ee36e26e8ef4444b965de210528b6132, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d66ad551af1f44558e8640231d60d6f9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f0e75a55959b4d10bfa0bb444fe4783e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/511d447291334dfba505f7b1a132e81b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/013a6676c88c40f9b6f177f11cd2b916, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/4709f33fcdb64c79a1884337edb6ade1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/b2c0ffc65cf44f5db971c087d82f54ae, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aed1ef2c2bb04b918da8d2557317da31, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/15f43fca6f8b445998527fa81d433856, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d27f7db0e95f4f2f8de15455d699ff6e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/616dbdf6af14451c9555be5d8efc03b8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/25b8b27df5f347dc94ff2f5abf83c4a3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/68c43aae920a43efaf3f1f8e42de1885, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/995330346b5148c494e38e665943a45b, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/18a2e509efad4307b8f96afa9eff7025, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/fb77f66cb9fe4b2ebb07caf860125c11, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/bbefc3594b1b400994475dd35a59c6f5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c123fd70b0b54aa3b99b43ec22182237, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aac1676d3b8e44a9b2074432a186680e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/597848dc40704995a5a3d8ffa075b0d2, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e16132aec7434407a3c7206019be2548, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c042e6487c0040d7a1321ec7a12c87f5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1f8f56a144124fe98ade943d92922299, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d47cdd06000f48bca53247ddc7d2e17c] to archive 2023-04-23 05:56:44,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:56:44,300 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e48e414f6c274416817ca2d081198e55 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e48e414f6c274416817ca2d081198e55 2023-04-23 05:56:44,301 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f4fd204e81954f1f88dd62d509898bd3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f4fd204e81954f1f88dd62d509898bd3 2023-04-23 05:56:44,302 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cd8c8e0871144720b98b118efb30325c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cd8c8e0871144720b98b118efb30325c 2023-04-23 05:56:44,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f841beaeb5f943a99ca59e612411dbcf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f841beaeb5f943a99ca59e612411dbcf 2023-04-23 05:56:44,304 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/46cc8b89c9d14031bf63ae031d41db44 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/46cc8b89c9d14031bf63ae031d41db44 2023-04-23 05:56:44,304 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/71f3f79d49324a7c8d2ca26359515087 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/71f3f79d49324a7c8d2ca26359515087 2023-04-23 05:56:44,305 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cc200a712c564041946e27a674c2e175 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/cc200a712c564041946e27a674c2e175 2023-04-23 05:56:44,306 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ea918a3838b34b89939f3bb3c2415f89 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ea918a3838b34b89939f3bb3c2415f89 2023-04-23 05:56:44,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/461d5f49a80d4af397555c1aa82aeb4f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/461d5f49a80d4af397555c1aa82aeb4f 2023-04-23 05:56:44,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/54dd6567b1dc45daa14b24ebd58c2e4f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/54dd6567b1dc45daa14b24ebd58c2e4f 2023-04-23 05:56:44,308 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/6c9f7e74c3c549528d25df6768ee8fb2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/6c9f7e74c3c549528d25df6768ee8fb2 2023-04-23 05:56:44,309 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/3decae56dcd54246b761fa6d254b5f81 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/3decae56dcd54246b761fa6d254b5f81 2023-04-23 05:56:44,310 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/07b3d5cff5794f3cbfe0d6ca35de6af3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/07b3d5cff5794f3cbfe0d6ca35de6af3 2023-04-23 05:56:44,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ee36e26e8ef4444b965de210528b6132 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/ee36e26e8ef4444b965de210528b6132 2023-04-23 05:56:44,311 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d66ad551af1f44558e8640231d60d6f9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d66ad551af1f44558e8640231d60d6f9 2023-04-23 05:56:44,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f0e75a55959b4d10bfa0bb444fe4783e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/f0e75a55959b4d10bfa0bb444fe4783e 2023-04-23 05:56:44,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/511d447291334dfba505f7b1a132e81b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/511d447291334dfba505f7b1a132e81b 2023-04-23 05:56:44,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/013a6676c88c40f9b6f177f11cd2b916 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/013a6676c88c40f9b6f177f11cd2b916 2023-04-23 05:56:44,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/4709f33fcdb64c79a1884337edb6ade1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/4709f33fcdb64c79a1884337edb6ade1 2023-04-23 05:56:44,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/b2c0ffc65cf44f5db971c087d82f54ae to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/b2c0ffc65cf44f5db971c087d82f54ae 2023-04-23 05:56:44,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aed1ef2c2bb04b918da8d2557317da31 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aed1ef2c2bb04b918da8d2557317da31 2023-04-23 05:56:44,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/15f43fca6f8b445998527fa81d433856 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/15f43fca6f8b445998527fa81d433856 2023-04-23 05:56:44,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d27f7db0e95f4f2f8de15455d699ff6e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d27f7db0e95f4f2f8de15455d699ff6e 2023-04-23 05:56:44,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/616dbdf6af14451c9555be5d8efc03b8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/616dbdf6af14451c9555be5d8efc03b8 2023-04-23 05:56:44,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/25b8b27df5f347dc94ff2f5abf83c4a3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/25b8b27df5f347dc94ff2f5abf83c4a3 2023-04-23 05:56:44,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/68c43aae920a43efaf3f1f8e42de1885 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/68c43aae920a43efaf3f1f8e42de1885 2023-04-23 05:56:44,320 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/995330346b5148c494e38e665943a45b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/995330346b5148c494e38e665943a45b 2023-04-23 05:56:44,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/18a2e509efad4307b8f96afa9eff7025 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/18a2e509efad4307b8f96afa9eff7025 2023-04-23 05:56:44,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/fb77f66cb9fe4b2ebb07caf860125c11 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/fb77f66cb9fe4b2ebb07caf860125c11 2023-04-23 05:56:44,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/bbefc3594b1b400994475dd35a59c6f5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/bbefc3594b1b400994475dd35a59c6f5 2023-04-23 05:56:44,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c123fd70b0b54aa3b99b43ec22182237 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c123fd70b0b54aa3b99b43ec22182237 2023-04-23 05:56:44,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aac1676d3b8e44a9b2074432a186680e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/aac1676d3b8e44a9b2074432a186680e 2023-04-23 05:56:44,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/597848dc40704995a5a3d8ffa075b0d2 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/597848dc40704995a5a3d8ffa075b0d2 2023-04-23 05:56:44,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e16132aec7434407a3c7206019be2548 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/e16132aec7434407a3c7206019be2548 2023-04-23 05:56:44,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c042e6487c0040d7a1321ec7a12c87f5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/c042e6487c0040d7a1321ec7a12c87f5 2023-04-23 05:56:44,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1f8f56a144124fe98ade943d92922299 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1f8f56a144124fe98ade943d92922299 2023-04-23 05:56:44,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d47cdd06000f48bca53247ddc7d2e17c to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/d47cdd06000f48bca53247ddc7d2e17c 2023-04-23 05:56:44,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9f2cc029d0074f3aa671d0cfd0a27695, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac5a59a919d14acf9e12fe33d1184e64, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/54101c3ecdf14aaa9fc86d7bf588bbb5, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/81df9bcee09e4232aef36513ccf1cbe6, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3a163f0ec3d44a4bbd6a808f24972624, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac9a7c7bad184c1f8fa25eaa1967e8cd, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/df0869251e8d43b691bad48b8eb57aa9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/18e4de8dc0d54a588df0dca188e54b6a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3aabf6d85ed647c99b07ad9cbb4afb63, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b9b8ea7a06cf4863aee638401a532da1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ddf0c0a0e3984026aa2a3a91f8334ad1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ee93f6ddb54343c8a06b71ada4a79931, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/626e79a2f62c49138809a4b76198868a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/339db8f12259461782b41f392bd0f5c8, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9599e9d7b7454c0c900b6f496a7ef167, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fe46b98c9fab4f07ab88560bec95e929, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e59fa9c210dc47e8be486af76699ee70, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/a1ab04f8eef3498b84225144d5307bf1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/c6e27f1777e44776b21c74363bc56128, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1db33ab66f924ab3babfdacb45a02d9a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/781050eeccfe4353b751f6f0effc33a9, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/7cceb233c8e24842811620e8898ced4f, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3fed367826d7470e9fe3683cd44f22d1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1e9db7dbb6604e79a0eebcdd14045b55, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/0eb178baeda24943b728250862ef1ac1, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/76c8a3d80b944482a7dc5748e33acc0a, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/334ac44ecdd14aaca051b639b6cd12c3, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/5b101c5cf8aa4bc2b997f5430905db35, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9617f6c2bc73462994f33a83901ee070, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fc7e1b0d92b046248214a9cee46c3b8e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b8554ec7e02e4d70abf07ff977b19891, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d523fca8f9994b2986b87d7522179433, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e446a50cbd5c4754b644e94d715c6e61, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d9de3f05dd6f4178a8cafd728166a417, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b0a1f5bb61aa471fa225b61c63ff1d0e, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/030874d814c54be8b4182a0c53e43ab3] to archive 2023-04-23 05:56:44,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-23 05:56:44,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9f2cc029d0074f3aa671d0cfd0a27695 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9f2cc029d0074f3aa671d0cfd0a27695 2023-04-23 05:56:44,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac5a59a919d14acf9e12fe33d1184e64 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac5a59a919d14acf9e12fe33d1184e64 2023-04-23 05:56:44,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/54101c3ecdf14aaa9fc86d7bf588bbb5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/54101c3ecdf14aaa9fc86d7bf588bbb5 2023-04-23 05:56:44,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/81df9bcee09e4232aef36513ccf1cbe6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/81df9bcee09e4232aef36513ccf1cbe6 2023-04-23 05:56:44,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3a163f0ec3d44a4bbd6a808f24972624 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3a163f0ec3d44a4bbd6a808f24972624 2023-04-23 05:56:44,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac9a7c7bad184c1f8fa25eaa1967e8cd to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ac9a7c7bad184c1f8fa25eaa1967e8cd 2023-04-23 05:56:44,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/df0869251e8d43b691bad48b8eb57aa9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/df0869251e8d43b691bad48b8eb57aa9 2023-04-23 05:56:44,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/18e4de8dc0d54a588df0dca188e54b6a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/18e4de8dc0d54a588df0dca188e54b6a 2023-04-23 05:56:44,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3aabf6d85ed647c99b07ad9cbb4afb63 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3aabf6d85ed647c99b07ad9cbb4afb63 2023-04-23 05:56:44,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b9b8ea7a06cf4863aee638401a532da1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b9b8ea7a06cf4863aee638401a532da1 2023-04-23 05:56:44,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ddf0c0a0e3984026aa2a3a91f8334ad1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ddf0c0a0e3984026aa2a3a91f8334ad1 2023-04-23 05:56:44,342 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ee93f6ddb54343c8a06b71ada4a79931 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/ee93f6ddb54343c8a06b71ada4a79931 2023-04-23 05:56:44,343 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/626e79a2f62c49138809a4b76198868a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/626e79a2f62c49138809a4b76198868a 2023-04-23 05:56:44,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/339db8f12259461782b41f392bd0f5c8 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/339db8f12259461782b41f392bd0f5c8 2023-04-23 05:56:44,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9599e9d7b7454c0c900b6f496a7ef167 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9599e9d7b7454c0c900b6f496a7ef167 2023-04-23 05:56:44,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fe46b98c9fab4f07ab88560bec95e929 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fe46b98c9fab4f07ab88560bec95e929 2023-04-23 05:56:44,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e59fa9c210dc47e8be486af76699ee70 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e59fa9c210dc47e8be486af76699ee70 2023-04-23 05:56:44,347 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/a1ab04f8eef3498b84225144d5307bf1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/a1ab04f8eef3498b84225144d5307bf1 2023-04-23 05:56:44,348 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/c6e27f1777e44776b21c74363bc56128 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/c6e27f1777e44776b21c74363bc56128 2023-04-23 05:56:44,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1db33ab66f924ab3babfdacb45a02d9a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1db33ab66f924ab3babfdacb45a02d9a 2023-04-23 05:56:44,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/781050eeccfe4353b751f6f0effc33a9 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/781050eeccfe4353b751f6f0effc33a9 2023-04-23 05:56:44,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/7cceb233c8e24842811620e8898ced4f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/7cceb233c8e24842811620e8898ced4f 2023-04-23 05:56:44,351 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3fed367826d7470e9fe3683cd44f22d1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/3fed367826d7470e9fe3683cd44f22d1 2023-04-23 05:56:44,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1e9db7dbb6604e79a0eebcdd14045b55 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/1e9db7dbb6604e79a0eebcdd14045b55 2023-04-23 05:56:44,353 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/0eb178baeda24943b728250862ef1ac1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/0eb178baeda24943b728250862ef1ac1 2023-04-23 05:56:44,354 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/76c8a3d80b944482a7dc5748e33acc0a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/76c8a3d80b944482a7dc5748e33acc0a 2023-04-23 05:56:44,354 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/334ac44ecdd14aaca051b639b6cd12c3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/334ac44ecdd14aaca051b639b6cd12c3 2023-04-23 05:56:44,355 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/5b101c5cf8aa4bc2b997f5430905db35 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/5b101c5cf8aa4bc2b997f5430905db35 2023-04-23 05:56:44,356 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9617f6c2bc73462994f33a83901ee070 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/9617f6c2bc73462994f33a83901ee070 2023-04-23 05:56:44,357 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fc7e1b0d92b046248214a9cee46c3b8e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/fc7e1b0d92b046248214a9cee46c3b8e 2023-04-23 05:56:44,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-23 05:56:44,359 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b8554ec7e02e4d70abf07ff977b19891 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b8554ec7e02e4d70abf07ff977b19891 2023-04-23 05:56:44,360 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d523fca8f9994b2986b87d7522179433 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d523fca8f9994b2986b87d7522179433 2023-04-23 05:56:44,360 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e446a50cbd5c4754b644e94d715c6e61 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/e446a50cbd5c4754b644e94d715c6e61 2023-04-23 05:56:44,361 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d9de3f05dd6f4178a8cafd728166a417 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/d9de3f05dd6f4178a8cafd728166a417 2023-04-23 05:56:44,362 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b0a1f5bb61aa471fa225b61c63ff1d0e to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/b0a1f5bb61aa471fa225b61c63ff1d0e 2023-04-23 05:56:44,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/030874d814c54be8b4182a0c53e43ab3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/030874d814c54be8b4182a0c53e43ab3 2023-04-23 05:56:44,367 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/recovered.edits/1025.seqid, newMaxSeqId=1025, maxSeqId=4 2023-04-23 05:56:44,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48. 2023-04-23 05:56:44,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9ad1ed5a4f4248a6e5f0e43c7f776f48: 2023-04-23 05:56:44,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:44,370 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=9ad1ed5a4f4248a6e5f0e43c7f776f48, regionState=CLOSED 2023-04-23 05:56:44,371 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682229404370"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682229404370"}]},"ts":"1682229404370"} 2023-04-23 05:56:44,372 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-23 05:56:44,373 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 9ad1ed5a4f4248a6e5f0e43c7f776f48, server=jenkins-hbase4.apache.org,40995,1682229223026 in 2.1120 sec 2023-04-23 05:56:44,374 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-23 05:56:44,374 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9ad1ed5a4f4248a6e5f0e43c7f776f48, UNASSIGN in 2.1160 sec 2023-04-23 05:56:44,374 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682229404374"}]},"ts":"1682229404374"} 2023-04-23 05:56:44,375 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-23 05:56:44,376 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-23 05:56:44,377 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1270 sec 2023-04-23 05:56:44,766 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-23 05:56:46,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-23 05:56:46,359 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-23 05:56:46,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-23 05:56:46,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:46,362 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:46,362 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:46,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-23 05:56:46,366 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:46,368 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C, FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/recovered.edits] 2023-04-23 05:56:46,370 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/72882dad7bc447eca9889e639b2719ef to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/72882dad7bc447eca9889e639b2719ef 2023-04-23 05:56:46,371 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/b8e8b75bce4c4da59e47553890bb4113 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/b8e8b75bce4c4da59e47553890bb4113 2023-04-23 05:56:46,372 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/de37c0980dff4dc093e95b191fc1b376 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/A/de37c0980dff4dc093e95b191fc1b376 2023-04-23 05:56:46,374 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/032fc9136dcd448ca4406369b314c209 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/032fc9136dcd448ca4406369b314c209 2023-04-23 05:56:46,375 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1e66424c976a479cbcda98fb9facd245 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/1e66424c976a479cbcda98fb9facd245 2023-04-23 05:56:46,376 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/71a7b132d6524799bec41a64d0da7e6a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/B/71a7b132d6524799bec41a64d0da7e6a 2023-04-23 05:56:46,378 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/259fe19f5e344db9b136495e6e35876f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/259fe19f5e344db9b136495e6e35876f 2023-04-23 05:56:46,379 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/4ba33c10dfed456aad98ddee2b3ca960 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/4ba33c10dfed456aad98ddee2b3ca960 2023-04-23 05:56:46,380 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/6d3375b9b09543b393a65883e782a70a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/C/6d3375b9b09543b393a65883e782a70a 2023-04-23 05:56:46,382 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/recovered.edits/1025.seqid to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48/recovered.edits/1025.seqid 2023-04-23 05:56:46,382 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/.tmp/data/default/TestAcidGuarantees/9ad1ed5a4f4248a6e5f0e43c7f776f48 2023-04-23 05:56:46,382 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-23 05:56:46,383 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-23 05:56:46,383 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-23 05:56:46,385 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304230950098d9e47453fa4ede6bea90798bf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304230950098d9e47453fa4ede6bea90798bf 2023-04-23 05:56:46,386 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233a329da295ed4045930b0dd7e2ddaa15 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233a329da295ed4045930b0dd7e2ddaa15 2023-04-23 05:56:46,387 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233f5522e0f86c401d8811ee4d5ceb58f3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304233f5522e0f86c401d8811ee4d5ceb58f3 2023-04-23 05:56:46,388 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042348771a579dac4d438ed2ed632bb5ae47 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042348771a579dac4d438ed2ed632bb5ae47 2023-04-23 05:56:46,389 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304235ffa0e1724474de78e0ba287f2138e07 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304235ffa0e1724474de78e0ba287f2138e07 2023-04-23 05:56:46,390 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236684d70bea304e319299eea39fe0d639 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304236684d70bea304e319299eea39fe0d639 2023-04-23 05:56:46,390 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042376dba5731a904730ac9fdaa629e0aa58 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042376dba5731a904730ac9fdaa629e0aa58 2023-04-23 05:56:46,391 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042379e8350ae3aa4af2953668c197b8d244 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042379e8350ae3aa4af2953668c197b8d244 2023-04-23 05:56:46,392 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304237abe59cbbe29455cbc712d711bbcfc1f to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304237abe59cbbe29455cbc712d711bbcfc1f 2023-04-23 05:56:46,393 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423821164724bb84866b98c04cc3ac9ce8a to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423821164724bb84866b98c04cc3ac9ce8a 2023-04-23 05:56:46,394 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304238cac9f1a7b0c468a8b6520eef528f958 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304238cac9f1a7b0c468a8b6520eef528f958 2023-04-23 05:56:46,394 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042395df11ba1c70417e972263d087959f8b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042395df11ba1c70417e972263d087959f8b 2023-04-23 05:56:46,395 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304239837e9a33d414d8bbfda4f29573d0707 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304239837e9a33d414d8bbfda4f29573d0707 2023-04-23 05:56:46,396 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a3e5e17975f14fcfab448adb1ba30989 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a3e5e17975f14fcfab448adb1ba30989 2023-04-23 05:56:46,397 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a7b1ff41622d49819c04b1be060f9ce5 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423a7b1ff41622d49819c04b1be060f9ce5 2023-04-23 05:56:46,398 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423b0a336854fcd49c9b47a8919c5e3dedf to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423b0a336854fcd49c9b47a8919c5e3dedf 2023-04-23 05:56:46,398 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423b99365350a74413fad22a17874a4e4f3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423b99365350a74413fad22a17874a4e4f3 2023-04-23 05:56:46,399 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423bc90b0d021a44d7089a29b8964d290d0 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423bc90b0d021a44d7089a29b8964d290d0 2023-04-23 05:56:46,400 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c5b6ef03dd2e4ba3afb1f32a8bd60431 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423c5b6ef03dd2e4ba3afb1f32a8bd60431 2023-04-23 05:56:46,401 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d111be43ebfe428fb0710d0db4951ce4 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d111be43ebfe428fb0710d0db4951ce4 2023-04-23 05:56:46,402 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d290a669a1814e22bce732dc74cfbf7b to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d290a669a1814e22bce732dc74cfbf7b 2023-04-23 05:56:46,402 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d372843221db4a64af581b3b27f1fa77 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d372843221db4a64af581b3b27f1fa77 2023-04-23 05:56:46,403 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d3e5ff1f4c7a4cc6be81eec272875c89 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d3e5ff1f4c7a4cc6be81eec272875c89 2023-04-23 05:56:46,404 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d4ac67cac6134873bc94b10d88402b39 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d4ac67cac6134873bc94b10d88402b39 2023-04-23 05:56:46,405 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d86eba10df0143c3a8fd565a1bf44eb7 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d86eba10df0143c3a8fd565a1bf44eb7 2023-04-23 05:56:46,406 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d96e2308e3ac4ef09429a07d2ae820b1 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d96e2308e3ac4ef09429a07d2ae820b1 2023-04-23 05:56:46,406 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d9a6c92cb0d041c286a4e430e3ef9f96 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423d9a6c92cb0d041c286a4e430e3ef9f96 2023-04-23 05:56:46,407 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423dd6d75715f8e4cb98c5b173f74760b93 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423dd6d75715f8e4cb98c5b173f74760b93 2023-04-23 05:56:46,408 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e1364125c91e4b17b86391bd67608912 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e1364125c91e4b17b86391bd67608912 2023-04-23 05:56:46,409 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e41b8997405c45e5b49a40ee314e53e6 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423e41b8997405c45e5b49a40ee314e53e6 2023-04-23 05:56:46,410 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f5c43122db964454b51f40ebe4fd9ba3 to hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230423f5c43122db964454b51f40ebe4fd9ba3 2023-04-23 05:56:46,410 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-23 05:56:46,412 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:46,414 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-23 05:56:46,415 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-23 05:56:46,416 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:46,416 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-23 05:56:46,416 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682229406416"}]},"ts":"9223372036854775807"} 2023-04-23 05:56:46,421 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-23 05:56:46,421 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9ad1ed5a4f4248a6e5f0e43c7f776f48, NAME => 'TestAcidGuarantees,,1682229379212.9ad1ed5a4f4248a6e5f0e43c7f776f48.', STARTKEY => '', ENDKEY => ''}] 2023-04-23 05:56:46,421 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-23 05:56:46,421 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682229406421"}]},"ts":"9223372036854775807"} 2023-04-23 05:56:46,423 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-23 05:56:46,425 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-23 05:56:46,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 65 msec 2023-04-23 05:56:46,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44901] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-23 05:56:46,464 INFO [Listener at localhost/41845] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-23 05:56:46,476 INFO [Listener at localhost/41845] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 228), OpenFileDescriptor=590 (was 562) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=405 (was 405), ProcessCount=169 (was 169), AvailableMemoryMB=12760 (was 12659) - AvailableMemoryMB LEAK? - 2023-04-23 05:56:46,476 INFO [Listener at localhost/41845] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-23 05:56:46,476 INFO [Listener at localhost/41845] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-23 05:56:46,476 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x164139d4 to 127.0.0.1:63802 2023-04-23 05:56:46,477 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:46,477 DEBUG [Listener at localhost/41845] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-23 05:56:46,477 DEBUG [Listener at localhost/41845] util.JVMClusterUtil(257): Found active master hash=1596708042, stopped=false 2023-04-23 05:56:46,477 INFO [Listener at localhost/41845] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:56:46,478 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-23 05:56:46,479 INFO [Listener at localhost/41845] procedure2.ProcedureExecutor(629): Stopping 2023-04-23 05:56:46,479 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-23 05:56:46,479 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:56:46,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-23 05:56:46,479 DEBUG [Listener at localhost/41845] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68e6aadc to 127.0.0.1:63802 2023-04-23 05:56:46,479 DEBUG [Listener at localhost/41845] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:46,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-23 05:56:46,479 INFO [Listener at localhost/41845] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40995,1682229223026' ***** 2023-04-23 05:56:46,479 INFO [Listener at localhost/41845] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-23 05:56:46,480 INFO [RS:0;jenkins-hbase4:40995] regionserver.HeapMemoryManager(220): Stopping 2023-04-23 05:56:46,480 INFO [RS:0;jenkins-hbase4:40995] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-23 05:56:46,480 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-23 05:56:46,480 INFO [RS:0;jenkins-hbase4:40995] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-23 05:56:46,480 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(3304): Received CLOSE for 4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:56:46,481 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:46,482 DEBUG [RS:0;jenkins-hbase4:40995] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x355c4049 to 127.0.0.1:63802 2023-04-23 05:56:46,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4c72146aba32a511e3acc1eb0b7832e6, disabling compactions & flushes 2023-04-23 05:56:46,482 DEBUG [RS:0;jenkins-hbase4:40995] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:46,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:56:46,482 INFO [RS:0;jenkins-hbase4:40995] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-23 05:56:46,482 INFO [RS:0;jenkins-hbase4:40995] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-23 05:56:46,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:56:46,482 INFO [RS:0;jenkins-hbase4:40995] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-23 05:56:46,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. after waiting 0 ms 2023-04-23 05:56:46,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:56:46,483 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-23 05:56:46,483 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4c72146aba32a511e3acc1eb0b7832e6 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-23 05:56:46,483 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-23 05:56:46,483 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-23 05:56:46,483 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 4c72146aba32a511e3acc1eb0b7832e6=hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6.} 2023-04-23 05:56:46,484 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-23 05:56:46,484 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-23 05:56:46,484 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-23 05:56:46,484 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-23 05:56:46,484 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-23 05:56:46,484 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-23 05:56:46,486 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1505): Waiting on 1588230740, 4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:56:46,495 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/.tmp/info/77da4cf641c74a2c8d01802b01c54a68 2023-04-23 05:56:46,500 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 77da4cf641c74a2c8d01802b01c54a68 2023-04-23 05:56:46,686 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1505): Waiting on 1588230740, 4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:56:46,785 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-23 05:56:46,785 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-23 05:56:46,886 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1505): Waiting on 1588230740, 4c72146aba32a511e3acc1eb0b7832e6 2023-04-23 05:56:46,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6/.tmp/info/7f4c1bcbc5684ff5ba51a4823a60f70a 2023-04-23 05:56:46,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6/.tmp/info/7f4c1bcbc5684ff5ba51a4823a60f70a as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6/info/7f4c1bcbc5684ff5ba51a4823a60f70a 2023-04-23 05:56:46,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6/info/7f4c1bcbc5684ff5ba51a4823a60f70a, entries=2, sequenceid=6, filesize=4.8 K 2023-04-23 05:56:46,905 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 4c72146aba32a511e3acc1eb0b7832e6 in 422ms, sequenceid=6, compaction requested=false 2023-04-23 05:56:46,906 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/.tmp/rep_barrier/142718d00b5f46dca084bdcfec37c01f 2023-04-23 05:56:46,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/namespace/4c72146aba32a511e3acc1eb0b7832e6/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-23 05:56:46,909 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 142718d00b5f46dca084bdcfec37c01f 2023-04-23 05:56:46,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:56:46,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4c72146aba32a511e3acc1eb0b7832e6: 2023-04-23 05:56:46,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682229226178.4c72146aba32a511e3acc1eb0b7832e6. 2023-04-23 05:56:47,087 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-23 05:56:47,287 DEBUG [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-23 05:56:47,315 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/.tmp/table/f1baeaedd127424a86e6b22567a8649b 2023-04-23 05:56:47,318 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f1baeaedd127424a86e6b22567a8649b 2023-04-23 05:56:47,319 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/.tmp/info/77da4cf641c74a2c8d01802b01c54a68 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/info/77da4cf641c74a2c8d01802b01c54a68 2023-04-23 05:56:47,322 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 77da4cf641c74a2c8d01802b01c54a68 2023-04-23 05:56:47,322 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/info/77da4cf641c74a2c8d01802b01c54a68, entries=22, sequenceid=93, filesize=7.1 K 2023-04-23 05:56:47,323 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/.tmp/rep_barrier/142718d00b5f46dca084bdcfec37c01f as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/rep_barrier/142718d00b5f46dca084bdcfec37c01f 2023-04-23 05:56:47,326 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 142718d00b5f46dca084bdcfec37c01f 2023-04-23 05:56:47,326 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/rep_barrier/142718d00b5f46dca084bdcfec37c01f, entries=6, sequenceid=93, filesize=5.5 K 2023-04-23 05:56:47,327 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/.tmp/table/f1baeaedd127424a86e6b22567a8649b as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/table/f1baeaedd127424a86e6b22567a8649b 2023-04-23 05:56:47,330 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for f1baeaedd127424a86e6b22567a8649b 2023-04-23 05:56:47,330 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/table/f1baeaedd127424a86e6b22567a8649b, entries=9, sequenceid=93, filesize=5.5 K 2023-04-23 05:56:47,331 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 847ms, sequenceid=93, compaction requested=false 2023-04-23 05:56:47,335 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-23 05:56:47,335 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-23 05:56:47,336 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-23 05:56:47,336 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-23 05:56:47,337 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-23 05:56:47,487 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40995,1682229223026; all regions closed. 2023-04-23 05:56:47,496 DEBUG [RS:0;jenkins-hbase4:40995] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/oldWALs 2023-04-23 05:56:47,496 INFO [RS:0;jenkins-hbase4:40995] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40995%2C1682229223026.meta:.meta(num 1682229225887) 2023-04-23 05:56:47,500 DEBUG [RS:0;jenkins-hbase4:40995] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/oldWALs 2023-04-23 05:56:47,500 INFO [RS:0;jenkins-hbase4:40995] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C40995%2C1682229223026:(num 1682229224938) 2023-04-23 05:56:47,500 DEBUG [RS:0;jenkins-hbase4:40995] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:47,500 INFO [RS:0;jenkins-hbase4:40995] regionserver.LeaseManager(133): Closed leases 2023-04-23 05:56:47,500 INFO [RS:0;jenkins-hbase4:40995] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-23 05:56:47,501 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-23 05:56:47,502 INFO [RS:0;jenkins-hbase4:40995] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40995 2023-04-23 05:56:47,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,40995,1682229223026 2023-04-23 05:56:47,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-23 05:56:47,507 ERROR [Listener at localhost/41845-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2cf67f81 rejected from java.util.concurrent.ThreadPoolExecutor@28c9fec7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2529] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-23 05:56:47,507 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-23 05:56:47,507 ERROR [Listener at localhost/41845-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4379c999 rejected from java.util.concurrent.ThreadPoolExecutor@28c9fec7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2529] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-23 05:56:47,508 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,40995,1682229223026] 2023-04-23 05:56:47,509 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,40995,1682229223026; numProcessing=1 2023-04-23 05:56:47,511 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,40995,1682229223026 already deleted, retry=false 2023-04-23 05:56:47,511 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,40995,1682229223026 expired; onlineServers=0 2023-04-23 05:56:47,511 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44901,1682229221800' ***** 2023-04-23 05:56:47,511 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-23 05:56:47,511 DEBUG [M:0;jenkins-hbase4:44901] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e16def1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-23 05:56:47,511 INFO [M:0;jenkins-hbase4:44901] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44901,1682229221800 2023-04-23 05:56:47,511 INFO [M:0;jenkins-hbase4:44901] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44901,1682229221800; all regions closed. 2023-04-23 05:56:47,511 DEBUG [M:0;jenkins-hbase4:44901] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-23 05:56:47,511 DEBUG [M:0;jenkins-hbase4:44901] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-23 05:56:47,511 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-23 05:56:47,511 DEBUG [M:0;jenkins-hbase4:44901] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-23 05:56:47,511 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682229224603] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682229224603,5,FailOnTimeoutGroup] 2023-04-23 05:56:47,512 INFO [M:0;jenkins-hbase4:44901] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-23 05:56:47,511 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682229224601] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682229224601,5,FailOnTimeoutGroup] 2023-04-23 05:56:47,512 INFO [M:0;jenkins-hbase4:44901] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-23 05:56:47,512 INFO [M:0;jenkins-hbase4:44901] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-23 05:56:47,512 DEBUG [M:0;jenkins-hbase4:44901] master.HMaster(1512): Stopping service threads 2023-04-23 05:56:47,513 INFO [M:0;jenkins-hbase4:44901] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-23 05:56:47,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-23 05:56:47,513 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-23 05:56:47,513 ERROR [M:0;jenkins-hbase4:44901] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-23 05:56:47,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-23 05:56:47,513 INFO [M:0;jenkins-hbase4:44901] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-23 05:56:47,514 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-23 05:56:47,514 DEBUG [M:0;jenkins-hbase4:44901] zookeeper.ZKUtil(398): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-23 05:56:47,514 WARN [M:0;jenkins-hbase4:44901] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-23 05:56:47,514 INFO [M:0;jenkins-hbase4:44901] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-23 05:56:47,514 INFO [M:0;jenkins-hbase4:44901] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-23 05:56:47,515 DEBUG [M:0;jenkins-hbase4:44901] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-23 05:56:47,515 INFO [M:0;jenkins-hbase4:44901] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-23 05:56:47,515 DEBUG [M:0;jenkins-hbase4:44901] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-23 05:56:47,515 DEBUG [M:0;jenkins-hbase4:44901] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-23 05:56:47,515 DEBUG [M:0;jenkins-hbase4:44901] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-23 05:56:47,515 INFO [M:0;jenkins-hbase4:44901] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.90 KB heapSize=393.74 KB 2023-04-23 05:56:47,525 INFO [M:0;jenkins-hbase4:44901] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.90 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/582029fc5a3441b3949e04053fccdd53 2023-04-23 05:56:47,528 INFO [M:0;jenkins-hbase4:44901] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 582029fc5a3441b3949e04053fccdd53 2023-04-23 05:56:47,529 DEBUG [M:0;jenkins-hbase4:44901] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/582029fc5a3441b3949e04053fccdd53 as hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/582029fc5a3441b3949e04053fccdd53 2023-04-23 05:56:47,532 INFO [M:0;jenkins-hbase4:44901] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 582029fc5a3441b3949e04053fccdd53 2023-04-23 05:56:47,532 INFO [M:0;jenkins-hbase4:44901] regionserver.HStore(1080): Added hdfs://localhost:44199/user/jenkins/test-data/0d4a5ead-8912-313c-32b4-d91137916965/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/582029fc5a3441b3949e04053fccdd53, entries=65, sequenceid=580, filesize=24.6 K 2023-04-23 05:56:47,532 INFO [M:0;jenkins-hbase4:44901] regionserver.HRegion(2948): Finished flush of dataSize ~344.90 KB/353182, heapSize ~393.73 KB/403176, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 17ms, sequenceid=580, compaction requested=false 2023-04-23 05:56:47,533 INFO [M:0;jenkins-hbase4:44901] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-23 05:56:47,533 DEBUG [M:0;jenkins-hbase4:44901] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-23 05:56:47,536 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-23 05:56:47,536 INFO [M:0;jenkins-hbase4:44901] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-23 05:56:47,537 INFO [M:0;jenkins-hbase4:44901] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44901 2023-04-23 05:56:47,539 DEBUG [M:0;jenkins-hbase4:44901] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,44901,1682229221800 already deleted, retry=false 2023-04-23 05:56:47,609 INFO [RS:0;jenkins-hbase4:40995] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40995,1682229223026; zookeeper connection closed. 2023-04-23 05:56:47,609 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-23 05:56:47,610 ERROR [Listener at localhost/41845-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3bb45e9c rejected from java.util.concurrent.ThreadPoolExecutor@28c9fec7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2529] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-23 05:56:47,610 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@71121311] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@71121311 2023-04-23 05:56:47,610 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): regionserver:40995-0x101af94c0060001, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-23 05:56:47,610 ERROR [Listener at localhost/41845-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5cafd1 rejected from java.util.concurrent.ThreadPoolExecutor@28c9fec7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2529] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-23 05:56:47,610 INFO [Listener at localhost/41845] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-23 05:56:47,709 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-23 05:56:47,709 INFO [M:0;jenkins-hbase4:44901] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44901,1682229221800; zookeeper connection closed. 2023-04-23 05:56:47,710 ERROR [Listener at localhost/41845-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7d69a535 rejected from java.util.concurrent.ThreadPoolExecutor@1f5156d4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4226] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-23 05:56:47,710 DEBUG [Listener at localhost/41845-EventThread] zookeeper.ZKWatcher(600): master:44901-0x101af94c0060000, quorum=127.0.0.1:63802, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-23 05:56:47,710 ERROR [Listener at localhost/41845-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@21579591 rejected from java.util.concurrent.ThreadPoolExecutor@1f5156d4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4226] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-23 05:56:47,712 WARN [Listener at localhost/41845] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-23 05:56:47,715 INFO [Listener at localhost/41845] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-23 05:56:47,823 WARN [BP-1628852212-172.31.14.131-1682229218324 heartbeating to localhost/127.0.0.1:44199] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-23 05:56:47,823 WARN [BP-1628852212-172.31.14.131-1682229218324 heartbeating to localhost/127.0.0.1:44199] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1628852212-172.31.14.131-1682229218324 (Datanode Uuid 4e0a1e55-43d6-49e6-9e0b-0c82420ce4fb) service to localhost/127.0.0.1:44199 2023-04-23 05:56:47,827 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/cluster_e3def1fe-aef0-d9be-2e96-db5ec2594b54/dfs/data/data1/current/BP-1628852212-172.31.14.131-1682229218324] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-23 05:56:47,828 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/dd505dd0-6cd5-8479-6066-27c7f5aeefa4/cluster_e3def1fe-aef0-d9be-2e96-db5ec2594b54/dfs/data/data2/current/BP-1628852212-172.31.14.131-1682229218324] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-23 05:56:47,874 INFO [Listener at localhost/41845] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-23 05:56:48,042 INFO [Listener at localhost/41845] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-23 05:56:48,211 INFO [Listener at localhost/41845] hbase.HBaseTestingUtility(1293): Minicluster is down